1 UNIVERSITY OF CALIFORNIA, SAN DIEGO ONLINE LEARNING ARCHITECTURE AND SUSTAINABLE BUSINESS MODEL TO SERVE LARGE SCALE AUDIENCES FOR THE UNIVERSITY OF CALIFORNIA A Capstone Team Project submitted in partial satisfaction of the requirements Of the Architecture-based Enterprise Systems Engineering Leadership Program Christine R. Bagwell Emily Deere Committee in charge: Professor Harold W. Sorenson, Chair Professor Ingolf Krüger Professor of Practice Alexander Zak Professor Thomas Roemer
3 The Team Project prepared by Christine R. Bagwell and Emily Deere is approved Hal Sorenson Chair University of California, San Diego 2013 iii
4 Table of Contents Signature Page iii Table of Contents... 4 Table of Contents iv Table Table of Figures of Figures... vi 6 Abstract Abstract... ix 9 1 Context Background UCWorld Project Financial Evaluation Problem Formulation Taxonomy Top Level Description Use Cases Security Requirements Managerial Levers Context Problem Complexity Risk Management UCWorld in the framework of Bounded Rationality Architectural Framework Basic Object Oriented Process iv
5 3.2 Selecting an Architectural Framework for UCWorld Viewpoints Architecture Design Tools Integrated Dictionary Solution Approach Architectural Description Security Models Executable Models Service Oriented Architecture Solution Implementation Pilot Requirements Decision Making Process Pilot Decision Next Steps Appendix #1 EDUCAUSE List of Questions for Universities v
6 Table of Figures Figure 1: Strategy defined by Function, System and Process 12 Figure 2: SWOT Analysis...16 Figure 3: McKinsey 7S Framework Figure 4: Strategy Diamond Figure 5: Leading MOOC Platforms Figure 6: NPV and NOV for an AESE Online Master s Program 29 Figure 7: Pro Forma Financials for an AESE Online Master s Program Figure 8: Taxonomy of Online Learning Terms..31 Figure 9: User Roles.34 Figure 10: IBM SOA Security Reference Model...49 Figure 11: Relevant Policies and Requirements Figure 12: DoDAF 2.03 Viewpoints for UCWorld..74 Figure 13: Capability Taxonomy for UCWorld Figure 14: Capability Phasing for UCWorld. 76 Figure 15: Capability Dependencies Diagram for UCWorld Figure 16: Conceptual Data Model for UCWorld Figure 17: Data Model for Learning Dimensions vi
7 Figure 18: High Level Package Diagram for UCWorld...80 Figure 19: Use Case #1 Learning Assets Figure 20: Learning Assets Sequence Diagram.. 85 Figure 21: Learning Assets Class Diagram Figure 22: Course Pre-Decomposition Object Diagram..87 Figure 23: Course Decomposition Activity Diagram Figure 24: Use Case #2 MOOC Cohorts.89 Figure 25: Cohort Groups State Diagram..90 Figure 26: Cohort Groups Activity Diagram. 91 Figure 27: Use Case #3 Learning Achievements...92 Figure 28: Learning Achievements Activity Diagram 93 Figure 29: Learning Achievements Sequence Diagram. 94 Figure 30: Triggered Achievements Sequence Diagram Figure 31: Use Case #4 Learning Opportunity Suggestions. 96 Figure 32: Learning Opportunity Class Diagram..97 Figure 33: Use Case #5 Learning Analytics..98 Figure 34: Learning Objectives Class Diagram. 99 Figure 35: Learning Analytics Sequence Diagram Figure 36: A Defined Learning Path Sequence Diagram vii
8 Figure 37: A Learning Objective Object Diagram Figure 38: Security Services Table Figure 39: Open ID Authentication Figure 40: Roles and Security Needs..105 Figure 41: Class Diagram for Security Roles Figure 42: SAML 2.0 Activity Diagram. 107 Figure 43: Authorization Services Diagram Figure 44: Integrity Services - Activity Diagram..110 Figure 45: Coloured Petri Net Queue Pattern Figure 46: Coloured Petri Net Example..114 Figure 47: Student Interface Diagram..115 Figure 48: Service-Oriented Architecture Figure 49: UC San Diego Single-Sign-On Diagram. 117 Figure 50: Composed Rich Services Figure 51: Enterprise Service Bus design for UCWorld. 119 Figure 52: Pilot Provider Comparison Chart viii
9 Abstract University of California, San Diego (UC San Diego) is a top research and teaching university that has traditionally concluded that online learning is not of equivalent value to traditional in person course instruction and thereby that it is not sufficient to uphold the academic standards for the institution. The educational landscape is changing with many universities creating and distributing massive open online courses (MOOCs). UC San Diego would like to deliver MOOCs in a manner that upholds UC San Diego s academic quality, effective learning outcomes, is economically sustainable and promotes research goals. The UCWorld online learning framework provides UC San Diego a path for establishing an integrated system with low startup costs that will align with UC San Diego s mission and vision. The specific strategic goals include increasing UC San Diego s brand recognition, enhancing UC San Diego s learning and research programs and creating revenue opportunities for the University. UCWorld extends the impact of UC San Diego research, generates grant and research opportunities and expands classroom resources. By creating an online learning platform that adheres to UC policies, utilizes UC San Diego infrastructure and systems as well as building on current open source technologies, UCWorld provides an opportunity for UC San Diego to generate revenues and establish a presence in online learning markets. ix
10 10 1 Context 1.1 Background UC San Diego (http://www.ucsd.edu) is a top research and teaching university that has traditionally concluded that online learning is not of equivalent value to traditional in person course instruction and thereby that it is not sufficient to uphold the academic standards for the institution. The educational landscape is changing with many universities creating and distributing massive open online courses (MOOCs). UC San Diego would like to deliver MOOCs in a manner that upholds UC San Diego s academic quality, effective learning outcomes, is economically sustainable and promotes research goals. 1.2 UCWorld Project A strong business case can be made to establish a MOOC-like platform that meets UC San Diego s rigorous academic standards, creates a sustainable business model and aligns with the University s strategic research interests beginning with graduate programs in Architecturebased Enterprise Systems Engineering (AESE) (http://aese.ucsd.edu) and other programs in the Jacobs School of Engineering (JSOE) (http://www.jacobsschool.ucsd.edu/), the San Diego Supercomputer Center (SDSC) (http://www.sdsc.edu/) and the Qualcomm Institute (http://www.calit2.net formerly the UC San Diego branch of the California Institute for Telecommunications and Information Technology or CalIT2). The platform will collect information from courses participants and make it available for open analysis by others. Information will include demographics, learning patterns, and learning outcomes for subject material and additional course data. Aggregation and analysis of the data can be used to further the research interests of the campus and create a Google-like ability to extrapolate meaning.
11 11 One major interest in the data is to understand more about how students learn, specifically in online environments; this is of particular interest to the project stakeholders. In this way, the project is a reflection of UC San Diego s considerable talent in research and data sciences Alignment with UC San Diego Strategic Goals UC San Diego is in the process of creating a new campuswide strategy, led by Chancellor Pradeep Khosla. UC San Diego s current mission statement is also in alignment with the project s strategic aim (http://chancellor.ucsd.edu/chancellor-khosla/letters/together-we-willdevelop-a-strategic-plan-to-define-our-future ): UC San Diego is dedicated to the advancement of knowledge through excellence in education and research at the undergraduate, graduate, professional school and postdoctoral levels. The campus is committed to community engagement, public service and industry partnerships in order to advance the health and well-being of our region, state, nation and the world. Our academic community of world-renowned faculty, bright students and dedicated staff is characterized by a culture of interdisciplinary collaboration and innovation which spans the globe. The proposed project aligns with UC San Diego s mission and strategy in the following ways: Offering MOOCs for credit at UC San Diego extends the reach of the university. The project will attract many customers (students) provided the courses are well selected and developed. It is imperative that technology choices and the architecture of the project account for flexibility needed to buffer disruptive technologies or to move to them. More thought needs to be given to the potential for making efficient use of university resources.
12 12 UC San Diego is a public service entity, so information delivered at reasonable prices could aid the educational pursuits of many students in the world. UC San Diego s research would span the globe and be viewed by top academics around the world. UC San Diego also strives to be an innovative and entrepreneurial organization. The MOOC enterprise architecture could spawn dozens of companies who would license the technology from UC San Diego. The content for the platform could be offered to many audiences including customized products for industry partners. This provides a mechanism for cultivating relationships with new student pools and employers open to covering tuition costs for their employees. This is of importance for programs, such as AESE, should they desire expansion of the student FTE served Strategy Evaluation Various techniques were used to evaluate a strategy for the UCWorld project. Context was examined in the form of function, systems and process. Standard methodologies such as SWOT analysis (strengths, weaknesses, opportunities and threats), Enterprise 7s Analysis (Bradach Jeffrey, 1996), 5 Forces and TOPS approach were used for analysis Context Function System Process Figure 1: Strategy defined by Function, System and Process
13 13 The first strategy was to look at context in the form of function, system and process. The context is to work with UC San Diego stakeholders, UC administration and other online course subject matter experts to create a case for enterprise business value for a MOOC-like platform Function The function of the system will create online courses that can be delivered to large audiences, optionally for credit, with data collected from the course participants. Pricing will be dependent on the ability to receive credit and/or will be for smaller dollar amounts to receive class only credit or only to participate in learning the material. Value of the system will be measured to ensure stakeholders are receiving the proper return on their investment. The platform will be developed iteratively, so that time to market delivery is shortened and early analysis can be used to improve the system for maximum return and effectiveness. The iterative function will be to evaluate the courses, evaluate pricing and understand the level of data to collect to perform analytics on. Data will be collected, analyzed and returned to researchers either to understand more about how students are learning and retaining information or for other research areas of interest for UC San Diego. Stakeholders will be consulted often to ensure their needs are being met along with keeping up with competition from other MOOC programs System The system will focus on integration of open source online systems and/or a combination of current cloud offerings. Learning modules will include lectures, course video, hands-on exercises, assessments and the latest MOOC features. The data analytics portion will be unique to the system and will include gathering data during the student s online learning
14 14 experience. Data such as pace of moving through the material, subject matter understanding, test scores, additional questions asked, demographics and more will be collected throughout each course. The system will require servers, data storage, data collection tools, online course creation tools, disaster recovery systems, backup tools and other mainstream information technology pieces Process The process will be the most complex piece of the strategy given that it will require numerous collaborators, administrative policies and potential bureaucratic hurdles. Faculty resistant to new learning modalities such as online courses can generate significant organizational entropy. The process will capture the key use cases needed by the platform and the requirements that flow from that activity. This will include subactivities such as mapping out the approval process for all for credit courses. Other subprocesses to include: course and learning asset creation processes, anonymization of data for open analytics platform, etc. Evaluation of learning outcomes and course effectiveness will be a key component. Once the first set of classes has been conducted, the strategy will need to be revisited and the cycle will begin again in determining the function that is needed, the systems to be built and the process to make it all work properly SWOT Analysis A SWOT analysis was performed to determine what issues need to be addressed throughout the project development. The strategy suggested by the SWOT (http://en.wikipedia.org/wiki/swot_analysis) analysis relates to UC San Diego s organizational strategy.
15 15 Strengths: characteristics of the business or project that give it an advantage over others Weaknesses: are characteristics that place the team at a disadvantage relative to others Opportunities: elements that the project could exploit to its advantage Threats: elements in the environment that could cause trouble for the business or project Strengths UC San Diego has a variety of missions that are synergistic with the online learning environment. UC San Diego has data centers, data storage and processing ability through the San Diego Supercomputer Center that can aid the project in moving forward. Major ORU/MRU directors are motivated to engage in online learning pilots. The open analytics and cohort-based designs will be unique to what other schools are doing and offer key differentiators and thus a competitive advantage. Opportunities The opportunity to create the first online learning data analytics system of its kind. Data could be used by researchers at UC San Diego or sold to others for a profit. Weaknesses UC San Diego is not traditionally a school that moves quickly in to these areas, a subset of faculty will hold up the progress given their fear of moving forward. The UC bureaucracy will halt the online system before it can get launched and/or it will be too expensive to overcome hurdles. If not designed iteratively and in appropriate phases, the amount of infrastructure to build for the platform and support infrastructure may be cost prohibitive. Threats Other MOOCs will have a greater competitive advantage before the UC San Diego platform is launched. Time to market for UC San Diego is a competitive disadvantage.
16 16 Could promote UC San Diego as an online learning leader and promote its individual research strengths to the world. Could bring a new population to UC San Diego for them to learn from, teach to and have as avid followers of their programs. A newer emerging technology could replace MOOCs and the system investment would not yield returns. The Chancellor s new strategic plan may not be synergistic with online learning aims, which in turn could remove support from the highest levels at UC San Diego. Figure 2: SWOT Analysis Enterprise 7 S Analysis It will be imperative to work closely with Chancellor Khosla, faculty leadership and administrators to bring all parties in to alignment on the overarching strategy. Collective disinterest from the university community for the MOOCs platform could hurt the project, add roadblocks and increase time to deployment. Working closely with key stakeholders will also be imperative in building success. Completing each segment of analysis for the 7S framework developed by McKinsey will ensure the project will meet the needs of the internal organization. Figure 3: McKinsey 7 S Framework
17 Structure As an institution of higher learning, UC San Diego has many facets to its structure from centralized and decentralized decision making and authority, to policy vacuums and obtuse power centers where process is not obvious. It will be important to determine the structure the MOOC should live within UC San Diego whether at an ORU level, the UC system level or an administrative level. For the best success, it would be recommended to start at the smallest level that would be a graduate audience within an ORU. It is recommended that the infrastructure grow from within the San Diego Supercomputer Center (SDSC) and pilot the AESE MAS curriculum, courses and learning assets that emerge from the Qualcomm Institute (CalIT2) and SDSC s new Data Sciences Institute (DSI). The platform should integrate with centralized systems managed by Administrative Computing and Telecommunications (ACT) with close coordination with the Registrar s Office Systems UC San Diego currently has all the pieces necessary to create an enterprise infrastructure. SDSC can host enterprise web-based software and databases and data storage providing disaster recovery and high availability. Several stakeholders have indicated interest in providing courses and content, both Academic Senate faculty and ORU directors. The data analytics and workflow to make analytics available does not yet exist, but talent exists at UC San Diego to create this including top computer scientists and data architects Shared Values UC San Diego is at its core is a research University. The MOOC platform s open analytics module could be used by researchers in their subject areas and by those interested in the cognitive, social and educational dimensions of online learning. Creating online courses does
18 18 not resonate with all faculty, since many believe that online learning lessens the importance of their work. This debate will have to continue until it becomes obvious that enough other Universities have entered the market that UC San Diego is falling behind Staff UC San Diego is comprised of 25,000 students and 25,000 staff, the same size as a small city in the United States. The community represents a wide and varied skill set that can be utilized to create and administer the enterprise architecture that in part exists in many areas around the campus Style The style of the project fits in well with an operational research unit (ORU) or the entrepreneurial spirit of Connect (http://connect.org/ ) or other UC San Diego organizations. Many students and staff would find this an engaging and exciting program to move forward with. Conservative or change averse faculty might be resistant to this new framework and teaching method Skills Ensuring UC San Diego properly markets this exciting opportunity within its own walls will be critical to the success as much as marketing the program to the outside community Forces Given the large number of MOOC offerings that were being developed when the project commenced, it was important to analyze the competitive market. Porter s Five Forces methodology was used; the strategy diamond determines what drives competitive intensity.
19 19 Figure 4: Strategy Diamond Threat of Entrants The barriers to entry are very low for MOOC providers. Courses are relatively easy to develop on relatively low cost platforms. Technology can be used to scale up with usage at a very low investment if usage is moderate. There is certainly a barrier as a MOOC provider wants to transition from a few courses to many. It is decidedly difficult to have a truly massive online course without brand name recognition and cache of some sort, such as a MOOC offered by Stanford and MIT. Large, renowned universities are most likely to be successful offering MOOCs and also most lack the flexible culture needed to be successful in an online business where technologies are often changing and customers expect a baseline of contemporary tools and approaches. New entrants can sometimes have an advantage of entering the market late as more mature technologies are more efficient, less costly or less bug prone. However, what is gained in technological simplicity is offset by the learning curve, albeit not insurmountable, about delivering a MOOC rather than a face-to-face (F2F) course. The learning curve (L%) for the first courses is very steep. It is difficult to recruit a critical mass of
20 20 staff needed for offering a MOOC program as there are not yet enough like programs to produce the talent needed Rivalry Among Incumbents Competition is low because few have entered the MOOC space with quality content. Rivalry is less defined by others offering MOOCs and arguably instead those providers with similar business models. For discussion s sake, as the economic model s definition is part of the project, the rivals are other universities offering credit for a fee; taking the course is otherwise free. Most MOOC providers are renowned higher education (higher ed) institutions. The courses offered depend upon faculty self-selecting more than an institutional strategy. In some cases, as with Stanford and MIT, the MOOCs offered are without competitors. For UC San Diego to enter this space, it would be important to ensure the MOOCs were strategically selected for niche strengths that are of interest to a wide population, e.g. MOOCs in introductory oceanography, bioengineering and data sciences. Where MOOCs abound in a field, such as Biology, UC San Diego could be successful differentiating by offering a course in biofuels specific subtopics of interest. MOOCs may require technologies not essential otherwise to the daily resident instruction. However, the adoption of such platforms may aid traditional instruction as supplements and therefore not constitute a barrier to exit Bargaining power This market force is less of a concern as a number of MOOC platforms are built upon open source software shown below. That does not mean the course platform is completely free. Other dependencies, such as database software or third party services to maintain the platform (very common) may require yearly agreements. Once one is committed to a technology, those suppliers may without warning raise the price dramatically effecting cost.
21 21 Another issue is that the provider might innovate and the new release may adversely impact quality without warning. One issue with offering any course online on a large scale is that the number of current, stable, scalable platforms is very few. Time to produce a new platform and to innovate a stable, scalable platform presents an opportunity cost. Bargaining power is not as great a threat for this proposal as it might be for a physical good, such as razors. Platform Name License Hosting Options Intellectual Property Restriction? Analytics Notes EdX Open Source Local or hosted No Yes, unsure of accessibility Coursera Proprietary Hosted Yes Limited Blackboard Proprietary Local or hosted Udacity Proprietary Hosted only No Yes With purchase of additional module based on per FTE student cost. Unknown Figure 5: Leading MOOC Platforms Threat of substitutes The threat of substitutes due to disruptive technologies is very great. New mashups and innovations are developed with reasonable regularity. While they might not constitute an advance in true quality, that is better achievement of learning outcomes ( better learning ), they often create a perception of increased value by the customer. The alleviating factor is that one
22 22 essential component of a successful MOOC is an excellent brand in a highered context. These institutions are not likely to be flexible and quickly adopt the disruptive technologies faster Thread/aid of complements Online courses in general have tremendous potential for complements with the current offerings at universities. UC San Diego in particular is very averse to offering online courses and one could argue has additional barriers intended to slow the synergies of sharing resources between F2F and remote instruction (RI) or MOOCs. Other public universities appear to have the same constraints whereas it is less present at private universities T-O-P Perspectives A T-O-P perspective provides the understanding of whether a project is worth going forward with. T-O-P looks at technical, organizational and personnel structures of an organization Technical UC San Diego seeks inquiry and evidence-based methods in choosing new projects; this stems from the institution s focus on research. Offering a MOOC platform and data mining/analytics project as a pilot that scales larger and in breadth is inline with the overall technical approach of the campus. Resources are available at SDSC for creating such a platform. This may also be a hindrance if contemplated by a centralized administrative unit that requires too much data before making a decision. Timing is critical and getting too far behind the curve of adoption is not recoverable at a certain stage.
23 Organizational UC San Diego is a knowledge-seeking and disseminating organization. Providing education to broader audiences and using the analytics to inform new programs and translational projects is aligned with the institution s mission. UC San Diego lacks the spontaneous will to change policies to enable online education; without concerted efforts and high level will, such as the California Governor s recent appeal to help launch the UC online course initiative, UC San Diego will have a difficult time launching such a platform as a large endeavor. More important than legislative backing, will be the voices of respected faculty encouraging their interested colleagues to try contributing course materials and/or to teach online. However, there is considerable support by individual faculty leadership and Senate members as evidenced by the attendance at multiple MOOC meetings. A grassroots approach from an organized research unit as SDSC or a partnership between SDSC and QI should be able to transcend the larger organizational distaste for online courses. The bodies below are concerned with the common themes of academic rigor, proper effort for credit hours earned, anti-plagiarism, timely completion and low attrition. Personnel UC San Diego staff, faculty and administrators have a surplus of intellectual rigor, experience branching into new areas. UC San Diego has the critical mass of interested parties to launch a MOOC pilot and to explore business models and architectures. Students on and off campus are very interested in MOOC offerings, especially from renowned institutions.
24 Analysis of The T-O-P Assessment Compared To The SWOT And 7 S Assessments The T-O-P assessment is more in favor of the MOOC project than the SWOT belied. The SWOT analysis exposed several deficiencies in everything from threats to diluting the organization s brand and value to inflexibility and lack of MOOC-compatible resources. It added the element of Technology to the analysis by suggesting a way to weave the research and educational interests of the university into the project to help secure funding from the stakeholders. When contrasted with the SWOT, the T-O-P finds many more paths of alignment with both UC San Diego s organizational and personal spheres including that the institution is full of knowledge seekers and disseminators. T-O-P and SWOT both identified places where inflexibility and being behind the curve of innovation could make UC San Diego not a viable place for the pilot project. The 7S was also critical of UC San Diego s flexibility, and as with the SWOT found more cases for pessimism about the project being derailed due to lack of institutional will, outdated policies and a lack of staff experienced with deploying and managing MOOCs. 1.3 Financial Evaluation Two financial models were considered for the project. The first was Net Present Value, which consists of a discount rate, initial investment amounts and the free cash flow generated from operations. The second Is Net Option Value, which takes in to account the valuation of a large project which requires multiple decision points to continue investing in the option or discontinuing further investment in the project Net Present Value The basis for calculating net present value is to understand the potential for creating future value with current investment dollars. In the context of the UCWorld project, the
25 25 following variables were used in calculating the future value V. In the context of V=(p-c v -c f /q) x q, the important drivers for this proposal include the following. For comparison, the drivers are contrasted with the value created in a face-to-face (F2F) course and a traditional distance (online) course. q the likely capacity of a MOOC is as the name belies, enormous. The hope would be that some courses would have 100,000 students rather than a typical undergraduate, lower division UC San Diego class that often has <500 students. q is 200 times greater than in a traditional model. c f the fixed cost of such a platform is relatively inexpensive and easily scales with demand, especially compared to the traditional model of classrooms in physical space. This model is much cheaper relative to traditional RI where the ratio of instructor to student is often 1:30. c v the initial variable cost is likely to be high. However, if shared resources between the university and the MOOC project are shared, it is possible to keep c v low. For example, usually a business would require paid leadership. As a university initiative it is possible to make use of existing staff and tenured faculty paid by other sources. A project completely dependent on technology has other advantages if it constantly innovates to streamline processes. This will also serve to keep c v low. UC San Diego could define pricing based on course and decide to charge a higher rate for courses with higher cost. p if resources are shared as described above in c v and c f is scaled with the demands and profit, the margin would likely be proportionally (%) large. The sales velocity of a MOOC is inherently very high. This will create opportunity to set the price where it is market competitive and profitable. UC San Diego could define pricing based on course and decide to charge a higher rate for courses with higher cost.
26 Net Option Value Real options in the form of net option values can be applied to assess different research and development alternatives associated with this project Financial Drivers The need for UC San Diego to look for additional revenue models. The importance of revenue to the campus Diminished California state funding Diminished pools of grant funding by both Federal and private Alumni and corporate donations needed Necessary to accept outside competition Online universities are profitable Many public universities are offering online courses and/or MOOCs Landscape is changing and UC San Diego cannot ignore disruptive technologies Using San Diego Supercomputer Center [SDSC] SDSC seeks to serve UC needs in a sustainable, self-supporting fashion to expand revenues SDSC is an expert in data management and analytics. SDSC has existing cloud services for elastic growth. SDSC already understands the security requirements for research endeavors. Promoting UC San Diego resources will increase adoption for online learning across campus
27 Step Function A way to start with pilot programs and move to the new systems when it is available. To add new programs (graduate/undergraduate/moocs) Start with profitable degree programs such as those at the Rady School of Management, the Jacobs School of Engineering Allow MOOCs to use Coursera or other platforms until UCWorld is online Move online undergraduate courses to UCWorld when it is available Decision Points Decision points for when investment in the UCWorld project needs to be re-evaluated are affected by many variables. Any of the variables could cause reevaluation of options investment: Enrollment Rates Inability to keep enrollment rates at estimated values Investment Costs Inability to manage costs to those provided by options investors Policy Change If the University of California changes its policies for online learning New Entrants to the Market If a new option enters the market that makes more economic sense
28 Financial Models for a Master s Program The Architecture-based Enterprise Systems Engineering master s program techniques were used to calculate real numbers for a profit driven program. Two complete sets of Pro Forma Financials based on the data produced for the AESE program were created. Based on these financials, both NPV and NOV to show how the initial investment for the AESE pilot yields a profitable model Assumptions Current AESE courses exist o o Traditional content investment has been made Current revenue model remains Current program continues 3-5 more years o o Approximately fifty students in-house per year Current Infrastructure continues Marketing and administration is handled by current AESE and JSOE staff Projected Revenues Year 1: 0 - AESE program used to create online courses Year 2: $500, students paying $10,000 each o Must still apply Year 3: $1,800, enrollments at $9,000 Year 4: $1,800, enrollments at $9,000 Year 5: $14,000, enrollments at $7,000 o Saturation Level
29 Conclusion With a $3.5 Million investment, revenue generation is created for UC San Diego and the AESE and Rady School online program that will bring an NOV of $40M for the next ten years. Figure 6: NPV and NOV for an AESE online master s program Figure 7: Pro Forma Financials for an AESE online master s program
30 30 By receiving an additional $2Million in grants and University funding, the NOV can be increased from $40M to $42M. The additional funding will also reduce risk for the program from internal funding sources Additional funding sources To find either initial funds or additional funds to sustain investment, outside grants and University funding can be pursued, such as: Chancellor support for MOOCs and online learning could create internal funding Grants for online learning are available to help fund internal operations Data analytics and inventions can be used to create new revenue. 2 Problem Formulation A top down development approach was used for system development. Utilizing iteration in the system develop model, more detail can be created as the problem formulation is expanded. 2.1 Taxonomy To understand many of the elements in this paper, a taxonomy was created. This will be a useful reference for readers to understand the basic concepts of MOOCs, online learning and specific learning terms. Category Term Description Example Potential Technologies Creation/ Authoring Software Software used to create learning assets. Might be an application or utility locally installed or a web-based technology. Captivate, Respondus and SoftChalk Potential Technologies Platform Also called a Learning or Course Management System (LMS or CMS). Might be locally hosted on a combination of web, BlackBoard, Coursera, EdX, Udacity
31 31 database and load balancing servers. Increasingly offered as hosted solutions. Potential Technologies Tool Web-based software package that extends or integrates with a platform. Piazza, Wimba Pedagogy Learning Outcome Evaluation of the degree of learning Administering pre and post tests with a learning asset and looking at the delta of improvement. Pedagogy Learning Goal A broad, abstract target for a student. Students should gain an appreciation for financial analysis Pedagogy Learning Objective A specific, measurable goal that a student should attain. A student will be able to calculate NPV and NOV. Pedagogy Asynchronous Interaction that does not occur in real time Discussion board, Pedagogy Synchronous Interaction in real time Chat, Video conference, phone Pedagogy Chunking Decomposing learning assets into lengths more compatible with the learning process. Pedagogy Cohort A group that matches criteria such as 'in same area', 'can study at same time', 'shares compatible characteristics for partnership' that one can team with to complete a course. A student joins a cohort based on interests, availability to work synchronously or as soon as waitlisted students reach critical mass to become a cohort. Figure 8: Taxonomy of online learning terms
32 Top Level Description The platform should align with UC San Diego s mission, increase the perceived value of UC San Diego to the local region and be aligned with UC San Diego s strategic academic themes. The MOOC platform should be developed thoughtfully factoring in its context at a top tier, research university (R1). Thus at its core be designed as an open system that exists to collect data and to be a subject of inquiry. The initiative should serve as a nexus for onlinebased curricular components such as learning modules, assignment models and resources that can be built into MOOCs on the platform if desired. Looking at functions, structures and processes from a problem definition viewpoint: Functions The platform exists to extend the reach of UC San Diego through an effective, inclusive learning platform. As a side benefit, the platform will deliver courseware that will be tracked and stored in a way that aids efficient datamining and open exploration of the greatest possible subset of data for anyone. Structure. The structure of the platform must be planned so that the data can be stored in a data warehouse model or using cloud-based data schemas. Data should be segregated appropriately, balancing system speed and easy querying with single sources of data attributes and the minimum user imprints (personally identifiable information and protected data). The platform should be open source and adhere to standards. Wherever possible, the platform should integrate technology components, tools and extensions. However, the number of components should be balanced with need for a secure system. For example, writing one s own modules and not publishing the code, rather than using open source, can make it more difficult to be compromised. Complexity should be lessened when possible by limiting the programming
33 33 languages and knowledge, skills and abilities needed by support staff to a core group. For example, a system that requires many different languages could be difficult or overly costly to staff. Processes. The platform will adhere to accepted programming methodologies as chosen by a systems architect. It is expected that a combination of Agile and traditional plan-driven methodologies will be used. Development needed for the deployment and management of the infrastructure and business components will tend towards the plan-driven, while the learning tools, sharing of modules and data mining will tend more towards Agile methods. A risk management strategy will be established at the outset of the initiative and revisited periodically. User focus groups, especially with students and faculty, will inform adjustments needed for early iterations of the overall platform and during roll outs of new features. Easy feedback mechanisms will be available for users. Regular monitoring of usage, satisfaction and other metrics will be reviewed by technical staff and an oversight committee of stakeholders Stakeholders and Users The system will be used by students, faculty, support staff, teaching assistants, designers, guests/auditors and administrators Stakeholders and Stakeholder Organizations University administration Faculty Directors of programs, others responsible for revenue and enrollments Legislators looking to show ROI for CA investment
34 Roles Role Students Auditors/guests Faculty Teaching Assistants Designers Support Staff Administration Function Take courses for college credit Take courses and units for fun, enrichment Teach online courses in addition to or in lieu of in person courses. Offer courses to an audience not usually included in the resident campus model. Able to access class and student information, teach subgroups, communicate with students and have access to a subset of faculty data. Professionals who create the class content by working with faculty, researchers and learning experts. Monitor student use and progress. Help troubleshoot issues. Enrollment, accreditation, transcripts, security Figure 9: User Roles User Requirements These requirements were collected from the stakeholders and user base. High availability for a global audience, multiple time zones High bandwidth ability to handle large amounts of traffic in spikes (midterms, assignment due dates, start of the term) including video streaming, podcasts, synchronous connection Real time failover for system o Constant saving of data for users (e.g. crash during test) Responsive design for any resolution and future devices o Adherent to relevant standards of accessibility Identity Management & Roles-based Authorization o Authentication of student credentials during tests
35 35 o o o Authentication of faculty and teaching assistants for grading Authentication of administration for transcripts/grades Feature rich, extensible design tools: synchronous communication/classroom interaction, knowledge bases o o Constantly expanding library of communication tools Mobile, texting, tablets and touch screens Administrative tools to set up/maintain courses for instructors, staff and designers Analytics for user behavior, outcomes o o o o Data Storage Needs Back-end records keeping of course and student information (secure). Analytics storage Reporting and dashboard tools Governance o UC System policies to approve courses for delivery and credit Academic Senate and other committees Accounting o Must be self funded, grant funded or stakeholder funded Financial systems to track admission fees, grant funds, fund indexes etc. Payments for designers, faculty, profits System Requirements System/Subsystem boundary Learning management system (LMS)/platform
36 36 o Course data Authentication system o Roles Identity Management Data Warehouse o o Student data Analytics Disaster Recovery / Failover systems o For availability Name major functions of the system o Registration system Faculty/Designers/Students create identity/roles o o Access to course materials Platform that connects tools and integrates other platforms Facilitate collaboration Assessment Interactive learning adaptive, custom Analytics collection o o o Tracks activity Tracks student learning habits (page hits, drop offs etc.) Tracks demographics Major functions of the system Registration system o Faculty/Designers/Students create identity/roles
37 37 Access to course materials Platform that connects tools and integrates other platforms o o o Facilitate collaboration Assessment Interactive learning adaptive, custom Analytics collection o o o Tracks activity Tracks student learning habits (page hits, drop offs etc.) Tracks demographics 2.3 Use Cases A use case model captures the requirements of a system so that users and other stakeholders understand how the system is being designed. Each use case will show a high-level view of behavior of UCWorld by someone or something outside of the system. A series of use cases were assembled that position UCWorld better than a traditional MOOC platform Use case #1: Learning Assets Use case defined: Instructors create and upload a sharable, discoverable learning asset. Goal In Context: Scope: Level: Create learning assets that can be used by others and that are categorized for easy discovery. Creation of the learning asset and associated metadata. Task Pre-Condition: User accounts must exist in identity management system. Users and roles needed must exist (and be available as data
38 38 feed). Metadata dimensions defined. Analytics are available for Faculty and Course Designers Success End Condition: Completed learning assets are placed in the MOOC Library and are available to be deployed as a component of a course or specified learning path. Minimal Guarantees: Primary Actor: Platform lacks proper classifications to add learning asset. Instructor or course designer misclassifies or disagrees with system administrator (or oversight function) on learning asset properties. Instructor or course designer Trigger Event: Faculty or MOOC decides to add a learning asset. Main Success Scenario New learning assets are added to the course library. Step Actor Action Description 1 Faculty Creates new learning asset. 2 Faculty Assigns metadata. 3 Administrator Approves learning asset for inclusion in the library. Scenario Extensions Step Condition Action Description 1a 2a Analytics exist on related topic Metadata options insufficient Faculty use analytics to determine what additional materials to create. To improve the outcome of the learning objective, analytics from past courses is used. Faculty requests new metadata options be added or changed. Scenario Variations Step Variable Possible Variations 3a Instructor or designer misclassifies metadata for learning Administrator returns the learning asset to the instructor/designer for
39 39 asset. confirmation/approval of changes or for added information. Related Information Schedule: Priority: Performance Target: Frequency: Super Use Case: Sub Use Case(s): TBD Must Critical On demand Create a specified path to achieving a learning objective. Administrator adjusts or expands metadata options. Librarian curates learning assets. Channel To Primary Actor: Secondary Actor(s): Channel(s) To Secondary Actor(s): Learning asset interface Administrator Learning asset creation interface Open Issues Issue ID A B C Issue Description Develop way to keep track of copyright information. Develop process for validating accessibility. Factor in the processes new content should trigger Use case #2: MOOC Cohorts Use case defined: Students sign up for MOOC cohort groups to facilitate synchronous learning experience. Goal In Context: Scope: Level: Students Register for courses and find a Cohorts Group that they will work in throughout the course. MOOC platform Task Pre-Condition: Courses exist that require cohorts.
40 40 Success End Condition: Groups need students (not all full). User accounts must exist in identity management system. Cohort is filled per defined requirements and students proceed as virtual cohort to start class. Minimal Guarantees: Primary Actor: Student cannot register in the course Incomplete information, such as no instructor to connect to new course. Course fails during deployment or delivery Student is unable to interact with Cohorts Student and student groups Trigger Event: Student decides to join a class and move through it with a group of students to maximize course impact. Main Success Scenario See Use Case Diagram for complete detail: Step Actor Action Description 1 Student Views available groups 2 Student Signs up for group 3 MOOC Platform Connects student to group. 4 MOOC Platform Decrements available slots in group. 5 MOOC Platform Checks counter, if zero, triggers cohort to begin. Scenario Extensions Step Condition Action Description 2a Student wants to change cohort group Will allow student to switch groups during course delivery 4 Student leaves group Counter in group moving to >0, causes availability if late start is allowed and within tolerance to add. Scenario Variations
41 41 Step Variable Possible Variations 2a. Student not in group Instructor or other role with access, assigns student to a group. Related Information Schedule: Priority: Performance Target: Frequency: Super Use Case: Sub Use Case(s): Channel To Primary Actor: Secondary Actor(s): Channel(s) To Secondary Actor(s): TBD Must Critical as a differentiator for this platform. As students enroll in course groups. Student takes course with cohort delivery. Cohort class begins. Dashboard with available groups. User that creates groups. MOOC management tools Open Issues Issue ID A Issue Description Need process and workflow for allowing students to form groups that span across different courses, e.g. students form regular study cohort and want to work together again Use case #3: Learning Achievements Use case defined: A student works towards and earns a learning achievement. Goal In Context: Create an Achievement System to keep students engaged in the MOOC Platform. Achievements are more granular than a degree or certificate and will include fun or easy items to encourage the student to make additional interactions within the system and work towards goals over time. Scope: Level: MOOC platform and identity management system Task
42 42 Pre-Condition: Account exists and is attached to at least one student role. Account contains minimum default profile information. Analytics Engine has access to student learning statistics from past courses. Analytics Engine has ability to store student interactions within the system including time based, click based and other activities. Success End Condition: Student can browse Achievement lists, select an achievement, track achievement progress and complete the achievement. Minimal Guarantees: System cannot store analytics needed to determine Achievement completion System cannot store Achievement history for a student Achievement list does not render properly Primary Actor: Trigger Event: MOOC platform s analytics engine Student logs in, a new Achievement is added or opportunity emerges, student performs an action that qualifies them for a whole or partial Achievement. Main Success Scenario See also Use Case diagram: Step Actor Action Description 1 Student Log in 2 Achievement List Shows student what Achievements are available and completed. <<can show progress>> 3 Student Performs system action: Course related (signs up, completes, takes a quiz) Cohort related (helps other student, signs up for Cohort) Training related (creates a sharable module, tutors a course) Statistical related (number of years in the system, times visiting certain areas) <<One will be selected for the use case diagram>>
43 43 4 Student <<Selects an Achievement that must be completed while part of a cohort group>> 5 Analytics Engine (AE) Records Student actions taken and creates a log of student activity along with associated statistics. 6 Achievement Module Reviews student statistics after an action has occurred to determine if a new Achievement is earned. 7 Achievement Module Updates Achievement List to show progress student has made. 8 Student Reviews Achievement List to see Achievement earned 9 Student <<Can share achievement with others>> <<Can select a name modifier if earned>> Scenario Extensions Step Condition Action Description 3a 3b 3c 3d Can show Achievement Progress A student can share an Achievement with others Can create a Student name modifier Can achievement cohort based Achievements If it takes 10 course completions for an Achievement, the Achievement list can show 8 course taken 2 needed A student s Achievements can be viewed by others in the system. They can see who else has achievements and use these to select desirable cohort groups. If a student gets a series of related Achievements, their username will get a modifier such as Worldly or Well Read Can select Achievements that are only available to earn when part of a cohort group. Related Information Schedule: Priority: Performance Target: TBD Medium distinguishing characteristic of MOOC platform. Student Engagement. Six months after pilot launch
44 44 Frequency: Super Use Case: Dynamic as new events are triggered Student Creates Account and Profile Sub Use Case(s): Channel To Primary Actor: Secondary Actor(s): Channel(s) To Secondary Actor(s): None at this time MOOC Platform Student Achievement List Use Case #4 Learning Opportunity Suggestions Use case defined: Platform dynamically builds, maintains and pushes learning opportunity suggestions and makes timely recommendations. Goal In Context: Scope: Level: Enable student to capitalize on learning opportunities in courses, groups and learning assets (chunks) that they have not used or aren t subscribed to. Keeps the student engaged in the MOOC system and gives information they can t get elsewhere to grow the subscription rate on the system. MOOC platform and identity management system Task Pre-Condition: Account exists attached to at least one student role. Account contains minimum default profile information. List of courses, learning objects and open groups and their attributes must exist. Opportunity engine is up-to-date with matches between users and relevant, new or courses not yet taken, learning assets or open groups. Analytics Engine has access to student learning statistics from past courses. Success End Condition: Student is presented with relevant suggestions for additional content or collaborations offered by the MOOC platform and profile is amended with more customized characteristic. Student sees how learning has progressed in courses and can utilize the information to make improvements in their performance. Minimal System has insufficient information to make any recommendations.
45 45 Guarantees: System cannot access student learning analytics Dashboard does not render properly Primary Actor: Trigger Event: MOOC platform s analytics engine Student logs in, new asset is added or opportunity emerges, student reaches new achievement or adds new learning characteristic Main Success Scenario Step Actor Action Description 1 Student Log in 2 Dashboard Collects notifications and current custom content for specified user. 3 Dashboard If pre-conditions are met, dashboard alerts student of new learning opportunities. 4 Student Hovers over each opportunity for more information or clicks for complete information. 5 Opportunity Engine Serves up additional information requested by client. 6 Analytics Engine (AE) Records information viewed and amends profile with new characteristics (if applicable). 7 Student Enrolls in new course, group or views learning asset. Scenario Extensions Step Condition Action Description 3a 3b No suggestions returned Prompts user to add more information/parameters to profile. Suggests most popular learning assets and courses. Suggests achievements of interest to student as challenges, gauntlets. Related Information Schedule: Priority: Performance Target: TBD High distinguishing characteristic of MOOC platform Six months after pilot launch
46 46 Frequency: Super Use Case: Sub Use Case(s): Channel To Primary Actor: Secondary Actor(s): Channel(s) To Secondary Actor(s): Dynamic as new events are triggered Student Creates Account and Profile Student Takes a Course Instructor creates course baseline for analytics. MOOC Platform Student Dashboard Open Issues Issue ID A Issue Description Connect opportunity suggestions to achievements appropriately Use case #5: Learning Analytics Use case defined: Course designers review analytics from users of learning assets and courses and make pedagogical changes based on that information. Goal In Context: Scope: Level: Use analytics to improve and enhance the learning experience for students. MOOC platform and analytics engine Task Pre-Condition: Learning assets exist. Analytics engine has usage statistics for learning assets. Success End Condition: Instructors and designers make improvements to courses based on analytics suggestions. Minimal Guarantees: Primary Actor: Trigger Event: System has insufficient information to make any recommendations. MOOC platform s analytics engine Student logs in, new asset is added or opportunity emerges, student reaches new achievement or adds new learning characteristic Main Success Scenario Step Actor Action Description
47 47 1 Instructor Designer Log in, access learning assets list and analytics notifications. 2 Dashboard Collects notifications and current custom content for specified user. 3 Dashboard If pre-conditions are met, dashboard alerts instructor/designer of new suggestions. 4 Instructor Designer 5 Analytics Engine (AE) Hovers over each opportunity for more information or clicks for complete information. Serves up additional information requested by client. 6 AE Records information viewed and amends profile with new characteristics (if applicable). 7 Instructor Accepts suggestion and confirms. Designer Scenario Extensions Step Condition Action Description 3a No suggestions returned Prompts instructor or designer to deploy learning asset and/or to add more links/paths to asset. Scenario Variations Step Variable Possible Variations 7a. Multiple instructors Asks or notifies co-instructors to confirm change. 7b Learning asset deployed in other courses Notifies other designers and instructors of suggested change acceptance and offers same. Related Information Schedule: Priority: Performance Target: Frequency: TBD Medium <Nine months after pilot launch Dynamic as new events are triggered
48 48 Super Use Case: Sub Use Case(s): Channel To Primary Actor: Secondary Actor(s): Channel(s) To Secondary Actor(s): TBD TBD MOOC Platform Instructor MOOC Platform Interface Open Issues Issue ID A B Issue Description Ensure that course designer class exists and appropriately scopes all activity related to course design decisions and dashboard notifications. Create path/model for stringing learning assets around a learning objective not tied to a course, e.g. Khan Academy topics. 2.4 Security Requirements Because UCWorld has such a disparate group of people accessing the systems, from administrators with access to grades and personal information, students who are trying to achieve their own goals and the general public always looking for holes in online systems UCWorld needs a security plan to match the defined use cases. To determine the requirements for security concerns, it was decided the IBM SOA Security Reference Model should be utilized. Each of use cases were reviewed and when security concerns were discovered, it was described in terms of the three classes: Business Security, IT Security, and Governance and Policy.
49 49 Figure 10: IBM SOA Security Reference Model Security Services BSS /ITSS/G & P Use Case McLaughlin Service Security Concern and Description 1 BSS Compliance and Reporting 1 BSS Trust Management & Concern: Courses will give credit that is not in accordance with accreditation policies. Examples include minimum course hours, assessment and course materials etc. Description: Ensure that courses are set up with the proper credit, degree accreditations and all necessary policies are followed. Compliance and Reporting ensures that proper accreditation policies are followed and report to show compliance to these policies. Concern: Faculty will have copyright materials stolen or proprietary course material stolen and used at other locations.
51 51 2 ITSS Authentication Services 2 Governance & Policy Governance and Risk Management o Profile information o Analytics collected during course delivery. Concern: That a student will set up a fake profile, or have someone else pretend to be them. When a student creates a profile, what proof exists that the student is who they say they are? The concern is that students can hire others to take courses for them. Description: Proper authentication is needed to make sure students are who they say they are to take courses, quizzes and tests. Systems that provide confidence of users identities are needed. Concern: With the large number of TA s, Faculty, Staff, that policies will allow too many people to view or modify student information, grades or faculty data. An example could be TA s who have access to student systems could cheat for friends or for profit. Governance should dictate who gets access to grading systems and during what time periods. Only TAs, Faculty and approved Administrators during the course delivery? Only certain Security personnel outside of course delivery? ALL ITSS Authorization Description: Data Governance needs to be put in place and policies followed that set and rescind data access in a regular and monitored environment. Proper access forms should be used and/or automated methods to keep data governance active. Concern: Students can use their credentials to view grade, exam or other information with their credentials. TAs or other employees get in to applications or data they should not have access to. Even with correct Governance on who should see data, and good Authentication to ensure people are who they say they are, systems have to be set up to ensure proper Authorization so that students can t get in to systems that they should not. Description: Per the governance and policies of the organization, what access does each user gain in the
52 52 2 ITSS - Confidentiality Services 2 BSS Non Repudiation Services 2 BSS Identity and Access system. Access will be a combination of Role, Course Access, Administrative rights etc. Concerns would be that someone is able to access a part of the application or data that they should not have authorization for. A student gets in to the course or grading administration, a TA can change grades in past student accounts etc. Concern: Outside entities could steal packets of data going in and out of the system during tests or exams and decode this data to turn in course material in their name. Description: Usage of technology such as SSL to ensure all data passing between users and applications is secure and cannot be sniffed by outside parties. Passing of grades and transcripts will need to have security so others cannot access and read them. Exam and quiz data should also be encrypted to ensure it does not hacked as it is transferred between the student and the TA/Graders. Concern: Students will not trust the systems and that data is secure and sent from the servers. Lack of credentials or certificates might lead a student to leave the MOOC platform. Description: In the system, assurance is needed that students are receiving tests or assessments and that the data received back will be from the student. The MOOC platform, should have proper Digital Certificates and PKI security enabled so that students know they are receiving data from the proper sender. Concern: Even with governance & policies that dictate who can see student course information and grades, users could still gain access to locations they should not have access to. Description: In the system, it is important to continually validate the roles of users. Failure to do so could cause users to continue having access to courses or student information they are no longer
53 53 3 BSS - Compliance and Reporting 3 ITSS Confidentiality Services entitled to have. o Employees separated from the MOOC o TAs no longer teaching or grading a course o Professors moving to different MOOCs To ensure the systems are sustainable, simple items like having students reset their passwords via their addresses and/or security questions should be set up. Concern: Failure to comply to a state or federal regulation will occur. Data could be incorrectly reported or distributed and remediation and notification would prove costly. Description: All policies should be considered and have assigned policy officers to ensure adherence to the policies. Policies affecting the UC system: FERPA standards for Student Information Reporting SB1386 Privacy Protection in California PII & SSN Privacy Policies (best to use outside financial/payroll systems) Credit Card Policies (best to use outside verification systems) Accreditation Policies for course credit, certificates or degrees PHI Disability Accommodations Concern: Data from student systems could be intercepted as it traverse the network systems. Data could contain PII information, credit card information etc. Description: Usage of technology such as SSL to ensure all data passing between users and applications is secure and cannot be sniffed by outside parties. Passing of grades and transcripts will need to have security so others cannot access and read them.
54 54 3 ITSS Identity Services ALL ITSS Authentication 4-5 BSS - Secure Systems and Networks Exam and quiz data should also be encrypted to ensure it does not hacked as it is transferred between the student and the TA/Graders. Concern: That a Student could gain access to more courses than the one that they are signed up for. Description: The roles and identity management systems also understand the current courses students are signed up for and only allow them to see these courses. Concern: Someone will use brute force to hack systems by looking for vulnerabilities in user accounts Description: Strong password requirements should be used as well as regular forced password updates and review password policies often. Concern: That students accessing systems use them in ways that they should not, i.e. storing large amounts of data, hacking servers and using them for their own purposes (streaming movies etc.). Description: Ensuring the networks do not get hacked and no malware is placed on the system. All Operating Systems and Applications are properly patched. No students use the servers to stream videos or disable systems 4 Governance and Policy - Policy Management Students trying to hack systems Intruders trying to get secure information from the servers Students using servers for malicious functions (stream movies, store music etc.) Concern: When recommending courses, faculty will think there is favoritism to recommend certain courses. Description: Algorithms will have to follow Governance Policies that ensure there is no favoritism to certain courses. IT Professionals will need to ensure that those creating code are not influenced by outside views and follow the
55 55 5 BSS Compliance and Reporting 5 Governance and Policy - Security Policy Management Governance & Policies of the organization. Concern: When Analytics are viewed, people could infer details about students given the demographic information. A combination of factors could allow someone to identify PII information. Description: When looking at policies, ensure that combinations of multiple pieces of information don t together constitute create PII. Concern: Only the proper access to the analytics information inside systems is possible when accessed by outside parties. That shadow systems are not created or data used improperly when access through services (Business Intelligence and reporting applications etc.) Description: Governance and policy will need to be created to determine who can access analytics data, use BI tools and download data for their own processing. 2.5 Managerial Levers A strategy for gaining buy-in for the projects needed for receiving funding and continuing momentum. To do this, an analysis of the stakeholders, scope, financial issues and a set of comprehensive benefits is needed. 1) Which stakeholders should be involved and do they have final decision making rights. What outside parties might need to be involved? 2) What is the scope of the project to present? This question includes: Build versus buy?, MOOC versus course credit?, Free versus a fee?. 3) What are the financial issues that must be solved?
56 56 This question includes understanding where initial funding for the project could come from, what the ownership rights of the project would be and how any revenue collected could be used. 4) The benefits that will be realized from the project include: o o o o o Marketing the University Bringing visibility to programs and professors Bringing education to the masses Additional revenue, enrollment, research power Analytics collected for research, pedagogy, learning improvement Composition For the project, there will be different layers of decision makers and interested parties who must be consulted. A responsibility assignment matrix will be created to properly define responsibilities, accountability and communication. A popular technique is a RACI chart, a technique used to determine who is Responsible for making each decision, who will be held Accountable for the decision, who needs to be Consulted with and who needs to be Informed of all decisions. Options include: UC System (UCOP, CA Senate, General Council) UC San Diego (Chancellor or central governance / Academic Senate) Rady School of Management and/or Jacobs School of Engineering Qualcomm Institute or SDSC Individual stakeholders (Ramesh Rao / Hal Sorenson/ Thomas Roemer/ Ingolf Krueger) Project owners (Emily Deere/ Christine Bagwell)
57 57 In order to make decisions on how to structure the project, the Project owners and stakeholders should be consulted. Key questions, including decision-making rights, project scope and funding streams, need to be decided. When key questions have been answered and documented, the specifics for the decision making plan can be determined. In order to gain approval for the project, all interested parties need to be identified, create responsibility assignment matrices and draft a communication plan. The University culture is one where most people and groups want to be consulted prior to a decision being made. Lack of consulting some parties can halt a decision making process, even if they were not part of the original project plan. 2.6 Context The UC System has an online learning initiative UC Online Education (UCOE) and other groups at UC San Diego are exploring MOOCs. A clear boundary needs to be established of where MOOCs will best fit on the campus. MOOCs could be developed for a particular department, graduate program or ORU and/or get funding through a grant program. Determining the boundary of the project will allow for completion of RASCI charts, communication plans and to proceed with the decision making plan. Given recent developments from Georgia Institute of Technology partnering with IBM for online graduate studies, funding from corporate sponsors is also a possibility. The larger the scope and boundary, the more decision makers and committees will need to be navigated. Creating a MOOC for UC San Diego versus building a MOOC for a particular department, graduate program or ORU will help create the level of decision makers that must be involved. If the scope or boundaries are too large, the project could get sidelined because
58 58 getting through all the legal matters, committees and academic senate issues would be too onerous. Deciding to accept money or give credit means that different committees will need to be added to the decision making process. Committees are used in the University structure to grant credit approvals, funding approvals etc Communication Different committees will need to be navigated, communication with current Technology Education Learning (TEL) groups and ensuring leaders on campus are aware of plans and decisions that are being made. All committees and leaders should be identified before the stakeholders and project owners move forward with final decision making processes. In order to achieve this, the communication plan should include weekly updates that are made available to all identified parties. As each committee is navigated or decision is made, it should be documented and shared with other decision makers. In the University setting, it is a good idea to create a website where people can obtain materials. SharePoint and newsletters can be used to distribute information to a large group of stakeholders Control When preliminary questions have been answered and the scope developed, the main decision makers should be identified and managed carefully. Many times decision makers have busy schedules and need to have questions prepared for them and possible outcomes presented as part of a discussion or meeting. Following a communication plan and finding any potential roadblocks will be important as well. If there is someone who does not support the project and they have influence on the
59 59 outcome, special care should be taken to meet with them and keep them informed of the benefits of the project Decision Process A large part of the decision making process will be to clearly define all questions, maintain responsibility assignment matrices for all decision makers and present multiple options to decision makers for discussion. The University culture is one where each concept or decision needs to be discussed repeatedly with different audiences and changes to decisions need to be allowed for. Where possible, stakeholders and decision makers will be engaged to adhere to decisions and allow the project to move forward without rework. As stated above, identifying any possible opponents early and meeting with them individually will be important. It would be undesirable for opponents to work behind the scenes or undermine any project work because they didn t feel consulted in the process. The decision making process will include multiple sessions with different groups where the same material will be covered repeatedly. As much as a more streamlined decision making process gives control to a single person and structures defined decision making sessions, the University culture does not allow for such a streamlined process. Multiple layers of decision making and bureaucracy must be navigated until each final decision maker has signed off. Where dissent and debate will be very healthy to the discussions needed, it is key to ensure it does not take the project off track. Since employment, work life and culture could change based on the creation of MOOCs at UC San Diego (whether at the campus level or departmental level) it is important to ensure all affected employees get to speak either at a town hall meeting or other venue. The final decision maker will have to know that not all parties can
60 60 be satisfied with a disruptive technology and have to stand firm in their decisions when emotions or dissent start to overpower the process Decision Outcomes The decision making process can be scored on how quickly buy in with the final decision makers and navigate all hurdles properly. Setting up the questions properly, understanding the RASCI charts and managing the process, will bode better than other methods, such as moving forward without properly communicating to all interested parties and being blocked later in the process Key Questions UC San Diego. Stakeholders are expected to ask key questions regarding the formation of a MOOC at 1) What are the risks of not moving forward with MOOCs or more aggressive online learning at UC San Diego? 2) What is the initial funding required? a) How many courses can be placed online for the initial funding? b) How many enrollments are expected from the initial funding? c) How will courses and professors be selected to be part of the first wave? 3) Who are the main competitors? a) What are the differentiating factors and are they important? 4) What are the risks of moving forward? a) Is there risk of taking away from regular enrollment levels? b) Will the organization s reputation be diluted?
61 61 c) Will learning standards be able to be maintained at a high level? 5) How will state legal issues, FERPA requirements, UCOP issues, Chancellor input, academic senate issues, departmental and college issues be navigated? See Appendix #1 55 questions an organization should ask regarding MOOCS https://docs.google.com/a/oakland.edu/document/d/1qjecixcymm_0qthc8al6zieqr14mnv46gphu82mc6ly/edit?usp=sharing 2.7 Problem Complexity Policy, Politics and Governance Complexity University of California is faculty-governed. For a large, unpopular shift to occur, there are very few or no mechanisms for circumventing faculty approval. Offering courses to non- UC, matriculating students and offering online, non-residential courses are both unpopular ideas that are stymied by restrictions in UC and UC San Diego policy. The disdain for offering online courses is reinforced by strict policies and sometimes seemingly haphazard application of the rules by Academic Senate committees. The governing structure is in flux and influential committees membership changes. In such a fluid state it is difficult to know who to involve and influence and what rules such a system needs to factor in. However, individual faculty retain great autonomy which is also an opportunity. Williams Ettouati s trailblazing act to deliver his Drug Discovery, Development and Commercialization course on the MOOC platform, Coursera illustrates this point. While the University has not created a clear path to offering MOOCs, Prof. Ettouati partnered with a third party to offer a unique course.
62 62 Name Type Notes Western Association of Schools & Colleges (WASC) Accrediting Body Accreditation status to an institution s ability to grant financial aid and whether or not the degree is recognized. UC Academic Senate UC San Diego Academic Senate Educational Policy Committee The Family Educational Rights and Privacy Act (FERPA) Governing body of academic policy for UC System Governing body of academic policy for UC San Diego UC San Diego Academic Senate Committee Federal Law Made up of representatives from each of UC s ten campuses. Made up of faculty representatives from the campus. The local campus committee that sets academic policies including remote instruction (distance/online learning). Governs privacy of student data. UC San Diego Registrar is the custodian of student data. Figure 11: Relevant Policies and Requirements Technical and Architectural Complexity A MOOC will provide courses to students in the tens of thousands of students. Another requirement is that students need to be able to self-register, no matter their affiliation with the institution. UC San Diego s current systems are neither designed for that scale or can accommodate non-affiliates. New systems and processes must be designed that work with the current environment and allow for a student s data to progress with them from nonaffiliate to affiliate status. Technologies for course delivery and back-end systems will change
63 63 during the architecture planning and design phase. The analytics piece will require large amounts of storage and compute power to data mine. Time, Moore s (http://en.wikipedia.org/wiki/moore%27s_law) and Kryder s Laws (http://en.wikipedia.org/wiki/kryder%27s_law#kryder.27s_law) are making this aspect less of an insurmountable technical issue. Moore s Law refers to the principle that computing power (performance of semiconductors) doubles every eighteen months. Kryder s Law refers to a similar concept that storage density increases even more quickly than Moore s Law. It s imperative that the system be designed to be flexible enough that the data will be useful and accessible, but also so that it is scoped so that privacy and other policies are accounted for, especially the Federal Educational Rights and Policy Act (FERPA). For translational projects and findings about learning to be extracted, faculty and research partners will need to be identified, polled for requirements and use cases they may not comprehend yet and then the project must be scoped for appropriate, roles-based access. 2.8 Risk Management The top ten potential risks for the project, together with impact, likelihood, transition indicators, and mitigation strategies are outlined: Leadership Shortfall Impact: if the stakeholders all serve as participants and none are able to devote time to a leadership role, the project could suffer from poor leadership. Likelihood: Moderately likely Transition indicators: Project targets are off schedule or worse, no dates have been set. Communication is poor and it s difficult to get the pulse on the project.
64 64 Mitigation strategies: the way to mitigate this risk is to identify a person who can serve in this role Inadequate Resources Impact: Low or no progress on initiative. Likelihood: Moderately likely, this is often a problem for university projects. Transition indicators: No software or few iterations are produced. Developer or their management ask more questions about funding than project requirements. Mitigation strategies: identify a stakeholder willing to provide minimal funds for a pilot and then upon pilot success, scale up commensurate with usage and demand Technology Changes Quickly, Project Left Behind Impact: technology advances and other MOOC platforms and schools will adopt faster than the project can keep pace with. UC tends to do a platform once and not revise often enough to keep up and the platform can become outdated. Likelihood: Moderately likely, this is often a problem for university projects. Transition indicators: New MOOC platforms emerge that have technology better suited to students needs. Customer requests often cite competitors capabilities. Mitigation strategies: Benchmark MOOC platforms and be alert to market. Assign point person to track and report back to team Student Adoption and Course Support/Design are Mismatched Impact: Students flock to the MOOC courses, but there is not adequate support for questions and there may be issues that affect the entire course. Or too few students use the platform for it to be considered a MOOC, let alone a successful online offering. Likelihood: Moderate to Extreme
65 65 Transition indicators: Usage analytics and enrollments show low utilization by customer or usage is high, but user satisfaction surveys are low. Complaints and support questions are high. Instructor enthusiasm is low. Instructors complain that the system doesn t work. Mitigation strategies: Offer courses with broad appeal not already duplicated by other MOOC providers of similar stature. Identify resources to assist instructors during the planning stages and to check in at regular intervals during the term for support. Train those monitoring other feedback loops to look for this risk and establish a reporting mechanism for bringing it to the attention of the project manager and others as appropriate Security Breach Impact: The platform is compromised either through technical or social hack that exposes Personally Identifiable Information (PII) and FERPA-protected data. Likelihood: Low to high, dependent on mitigation strategy, resources and attention Transition indicators: Accepted security protocols for detecting security breaches. Mitigation strategies: Engage system architects experienced in factoring reasonable security measures into their architectures. Allot appropriate resources to a security team. Include security team in reviews of system architectures, being mindful that security can t be the primary goal once compliance is met No Use of Analytics or Not Usable Impact: No faculty, researchers or others make use of the wealth of analytics. There is so much data that is not properly organized, that it can t be used. Likelihood: Extremely high Transition indicators: No activity in datamining. Mitigation strategies: Create coherent data architecture factoring in later reuse and
66 66 datamining. Make a very generic, redacted, non-sensitive data available to an online, nonaffiliate audience Platform Implodes Impact: The MOOC platform built, extended or contracted with fails to perform and is a fundamental failure. Other interdepencies such as networking could be overwhelmed to capacity. Likelihood: Low, largely because the project team and key stakeholders have a wealth of experience in this area. Transition indicators: The system is often down. Mitigation strategies: Create content that is platform agnostic. Have a backup plan for an alternate platform or delivery model should the chosen system not function properly or perform well under load. Assign role of watching this carefully and to someone specifically. For issues like network bottlenecks, have a geographically dispersed platform Scope Creep Impact: In order to satisfy all key stakeholders or due to many possible factors including over zealousness, the project includes too many features and modules. Likelihood: Low to Moderate Transition indicators: Progress on deliverables is low. Developers are confused about where to begin. First software artifacts are found to not factor in all components. Mitigation strategies: Plan the project in phases dependent on completion or near completion of earlier projects Academic Senate, Faculty or Accreditation Body Blocks or Slows Down Project and/or Senior Management Resistance
67 67 Impact: Senior management resistance is a likely organizational barrier. Senior management at UC and UC San Diego, is the true governing body, the faculty senate. They control the policy as well as the resource pool needed to provide content. The faculty comprise the Academic Senate. The idea of accreditation bodies such as WASC or ABET pulling or suspending accreditation is very threatening and could shut UC San Diego or a program down. Likelihood: Unable to determine (Low to High) Transition indicators: Notice from a Senate committee, Dean or similar to stop offering online courses or policy changes crucial to the initiative stall or are passed with unhelpful policies. ABET or WASC suspends accreditation or requires more rigorous reporting or requires a shorter interval before the next review period. Administrators weep openly at WASC s mention in meetings. Mitigation strategies: As online courses are developed, document elements required by the Senate and accrediting bodies such as Learning Goals and Objectives, assessment and antiplagiarism and cheating reducing strategies. Shore up support before launch, especially with the Chair of the Senate, Educational Policy, the Associate Dean of Undergraduate Education and other appropriate Deans. An AESE professor made an observation that other top tier research institutions don t face this organizational hurdle, such as Stanford and MIT. It is not realistic to think that governance model is mutable. However, it would be useful and is likely to be successful, to shore up support from the UC San Diego Chancellor, the President of the Academic Senate and the Chair of the Educational Policy committee that drafts policy in this realm.
68 Faculty Participation Low Impact: Faculty are unwilling to design and teach MOOCs and are not willing to share their materials for re-use by other faculty. Likelihood: Moderate to Extreme Transition indicators: Few courses are in the pipeline to be taught on the MOOC platform. The website area for re-use of exemplar modules is nearly empty. Mitigation strategies: Enlist support of interested faculty before launch. Adequately formulate user requirements with the instructor as a user type. UC San Diego has enough interest to launch a MOOC effort and success measured in usage and increased brand currency should drive additional faculty towards MOOCs. Administrators, especially Deans and Chairs should examine ways to incentivize faculty to teach MOOCs, such as giving teaching release for MOOCs and monetary incentive to develop MOOCs. Based on this risk analysis, there are pieces of the project that should be plan-driven and others that lend to agile development process. The project s primary goal is to respond to change and to add rapid value. The environment is project-focused and ever changing. Though the customers wouldn t be on-site for the development necessarily, it would be easy to involve students and faculty throughout the development process. The development should include a simple design at first, followed by short increments refactoring. The pieces that should be Agiledriven include reporting and analytics, dashboards and learning tools. The infrastructure pieces that have stable requirements and are dependent on solid, production quality environments are security, identity management, network design, servers and maintenance (patching), database design and architecture and the service bus.
69 UCWorld in the framework of Bounded Rationality The three areas of limited information, processing resources and cognitive limitations are all inter-related in the context of UCWorld and bounded rationality. An instance for each and a discussion of potential ways to alleviate this limitation are as follows Limited information Limitation: Governing bodies, such as the Academic Senate, opt for gathering additional information rather decisively entering into online courses. Faculty and the governing body of the University, the Academic Senate, are loath to expand on or allow for online course delivery. A reason often cited is a lack of information that these programs are effective or as effective as in classroom experiences. When research proving the efficacy of such programs is presented, the common response is that UC San Diego is special and the analysis must include UC San Diego-specific programs, faculty and students for true comparisons. Through the use of MOOCs, faculty can potentially hold their studies of how well students can achieve desired learning outcomes in a very accelerated fashion. Faculty can post much of the same materials including videos, Powerpoint, text and ask questions as they would in class on the discussion board. Students can be given the same assessment instruments, such as quizzes, and then outcomes can be compared. With the scale of MOOCs and their ad hoc nature of being offered on demand, the experiment to sample and compare learning outcomes using UC San Diego materials can be done much quicker Cognitive limitations Limitation: Gathering and analyzing data to prove effectiveness and efficacy of a learning outcome is complex and not easily accomplished with current learning tools available. UCWorld works to compensate for bounded rationality in the project design itself. It is difficult
70 70 for an individual instructor or administrator to simultaneously to draw conclusions or capture trends from the data artifacts collected by a learning management system (LMS). Assessment information can yield results on how well a specific learning goal was achieved, but it is difficult to factor in learning style and performance for any particular learning module. As well most LMS platforms don t contain basic demographic information usually held in identity management systems, such as educational interests and goals, personality or learning style assessments. This is even more complex when delivered to a MOOC audience that can number in the 1000 s of students per course. UCWorld s analytics engine and enterprise architecture design plan for the use case of an instructor analyzing many different types of data. It extends the idea to make the usage and performance data, or analytics, available in a deidentified (anonymized) version so any data scientist, educator or layperson with basic data analysis skills can make use of the analytics Limited (processing) resources Limitation: The intersection is small between those who could make use of learning assets (courses) and analysis of learning activities and those with adequate resources to obtain and process such data. The sample sizes of the MOOC courses are too large to process with crude brain power and too complex to reason via intuition alone. Success of MOOC providers/comparators, Coursera, Udemy and EdX, shows significant untapped demand for learning opportunities. None of the comparators allow for completely open analysis of the data collected by the MOOC. The UCWorld platform will offer a limited set of web-based tools to allow for simple processing of data for analysis. Users and faculty will have access to download larger datasets with tutorial links for further analysis as well as documented advice describing the data sets and key fields recommended
71 71 3 Architectural Framework An enterprise is a large and complex system of systems. To manage the complexity of the system, an architectural framework is used by enterprise architects. The architectural framework provides tools and methods to focus on different viewpoints to match the needs of the stakeholders within the organization. UCWorld will base its architectural design methodology on an Object Oriented (OO) architectural development process. 3.1 Basic Object Oriented Process The description of the process for designing a DoDAF conformant architecture using OO is presented in four parts: Part 0: Describe a basic OO process for Architecture Design and examine DoDAF requirements based on the key entities and their relationships Part 1: Part 2: Parts 3A : and 3B Establish and Describe the Requirements: Operational Concept and Requirements; Use Cases and Diagrams Develop Logical Architecture Description: Class Diagrams, Behavioral Diagrams, Rule Model, Concordance Develop Physical Architecture Description: Physical Nodes and Links; Component Diagrams, allocation to Physical Nodes and Links (Deployment Diagram) All Parts: Maintain Integrated Dictionary
72 Selecting an Architectural Framework for UCWorld UCWorld is a learning platform that provides MOOC functionality and extends the concept to include open analytics, cohort-based MOOCs and online gaming-style achievements that scale in sustainable, novel financial model(s). The architecture will solve the following problem. Current MOOC platforms are merely rebranded Learning Management Systems (LMS) and don t provide tools for students and instructors to reuse analytics easily for better learning outcomes. The learning platform will expand on current MOOC functionality as well as incorporate Khan Academy style delivery of knowledge chunks, discrete learning paths and reuse and sharing of learning assets. For the UCWorld project, the Department of Defense Architectural Framework 2.02 (DoDAF was selected. DoDAF 2.02 is a top down and spiral methodology with an emphasis on concordance. A layered approach starting at the stakeholders and use cases will be used and will drill down to more specific design processes. The operational views will then be created and then test as appropriate using executable architectures. It is imperative that concordance between the different viewpoints and architectural elements be maintained. 3.3 Viewpoints An architecture should be developed by using iteration and a process of refinement. A major purpose of this is to make sure detailed work does not commence prior to understanding the high level requirements. It is also used to make ensure completed viewpoints and designs are reviewed and meet new functional requirements that have been uncovered during the process and redesign any systems before development ever takes place. DoDAF 2.03 provides a set of viewpoints with corresponding definitions and tools for each viewpoint:
73 73 All Viewpoint (required) (AV) Project Viewpoint (PV) Capability Viewpoint (CV) Services Viewpoint (SvcV) Data and Information Viewpoint Standards Viewpoint (STDV) (DIV) Systems Viewpoint (SV) Operation Viewpoint (OV) Models The process of refinement: Start at the highest level (conceptual): Capabilities and Operational Concept The project team needs to understand the mission of the organization and the problem(s) that the project aims to address. At this level, all user needs must be understood as well as the performance level required from the stakeholders. The team should scope the architecture and plan the project Move to a high level (logical): Operational and Data Viewpoints At this level the functional requirements are created and iterate through a process of creating the services, systems and data viewpoints. Hire the right team To the Intermediate level: Services, Systems and Data viewpoints The project team will create a full set of UML diagrams to relate the services, systems and data to the different systems. Emphasis will be placed on making sure legacy systems are represented in the diagrams.
74 And final to the lower level: System component design Teams will build individual systems based on the diagrams and designs delivered to them DoDAF 2.03 Viewpoints UCWorld will use the DoDAF framework v2.02 for the architectural development process. The following viewpoints use a layered approach and will lend to a comprehensive architecture. To begin the process, the Capability and Data and Information viewpoints will be defined. Other layers will be defined as the use cases are further defined. CV-1: Vision CV-2 Capability Taxonomy CV-3 Capability Phasing CV-4 Capabilities Dependency Diagram DIV-1: Conceptual Data Model DIV-2: Logical Data Model DIV-3: Physical Data Model OV-1: High-Level Operational Concept Graphic OV-2: Operational Resource Flow Description OV-3: Operational Resource Flow Matrix OV-4: Organizational Relationships Chart SV-1 Deployment Diagram Figure 12: DoDAF 2.03 Viewpoints for UCWorld CV-1 Vision UC San Diego is a public University that is world renowned for its research and teaching. Faculty at UC San Diego have traditionally felt that online learning would not be sufficient to uphold the academic standards for the institution. The project s goal is to create
75 75 and host a system to develop and deliver MOOCs that uphold the academic quality of the University, lends to the research and teaching missions and is economically sustainable. Open analytics Cohort groups Achievement system Opportunities engine CV-2 Capability Taxonomy A capability taxonomy is architectural data repository with definitions of all terms used throughout the architectural data and presentations capability Display <<capability>> <<Dashboard>> capability Analytics <<capability>> <<Tracking>> <<Opportunitie s Engine>> <<Achievements >> capability Authentication <<capability>> <<Roles>> capability Content Repository <<capability>> <<Creation>> capability MOOC Platform <<capability>> <<Course Delivery>> <<capability>> <<capability>> <<capability>> <<Learning Styles>> <<Assignments>> <<Groups & Collaborations>> <<capability>> <<capability>> <<capability>> <<Profiles>> <<Learning <<Prescribed Dimensions>> <<capability>> <<Curation>> <<capability>> <<Rights Management>> <<capability>> <<Standards>> Paths>> <<capability>> <<Assessment & Grading>> <<capability>> <<Rules>> Figure 13: Capability Taxonomy for UCWorld capability ESB <<capability>> <<Monitoring>> <<capability>> <<Logging>> <<capability>> <<Federating>> CV-3 Capability Phasing Capability phasing is the planned achievement of capability at different points in time or during specific periods of time. The CV-3 shows the capability phasing in terms of the
76 76 activities, conditions, desired effects, rules complied with, resource consumption and production, and measures, without regard to the performer and location solutions Exists Q1 Q2 Q3 Q Q1 Capability A. Display A.1 Dashboard B. Analytics B.1. Tracking B.2. Opportunities Engine B.3. Achievements C. Authentication C.1. Roles C.2. Learning Styles Complete C.3. Profiles D. Content Repository D.1. Creation D.2. Assignments D.3. Learning Dimensions D.4. Curation D.5. Rights Management D.6. Standards Complete Complete E. MOOC Platform E.1. Course Delivery E.2. Groups & Collaborations E.3. Prescribed Paths
77 77 E.4. Assessment & Grading E.5. Rules F. ESB F.1. Monitoring F.2. Logging F.3. Federation Figure 14: Capability Phasing for UCWorld CV-4 Capabilities Dependency Diagram Capabilities dependency diagrams show the dependencies between planned capabilities and the definition of logical groupings of capabilities. Figure 15: Capability Dependencies Diagram for UCWorld
78 DIV- 1 Conceptual Data Model The required high-level data concepts and their relationships DIIV 2 Logical Data Model Figure 16: Conceptual Data Model for UCWorld The Logical Data Model is the documentation of the data requirements and structural business process (activity) rules. The next chapter shows complete class diagrams for the UCWorld. The data requirements learning dimensions are shown below.
79 DIV 3 Physical Data Model Figure 17: Data Model for Learning Dimensions entities. The Physical Data Model is the physical implementation format of the Logical Data Model
80 Architecture Design Tools Figure 18: High Level Package Diagram for UCWorld The architecture description language and tools to be used are UML 2.0 modeling completed in Enterprise Architect (EA) and the executable model will be constructed in Colored Petri Nets using CPN Tools. Business process modeling will be completed with the BPMN add-ins for EA. Models from SOA will also be used.
81 Integrated Dictionary To maintain integrity of the systems and services throughout the process, a set of glossaries of terms for each viewpoint to use were created. Glossaries include the entity classes, value classes and service classes that will make up the main architectural elements Entity Classes Class Name Description Administrator Need sub-role for administering system Auditor Subclass of role needed for misc. users of system that don't need grades and whose activities should not be factored into analytics, e.g. accreditation body, department coordinator Cohort Class to describe type of course based on having a critical mass to begin, rather than a specific date Contact Dimension of demographics for analytics engine, opportunity engine Information Course A defined topic to achieve a set of learning goals, often recognized for credit or certificate progress Course Subclass of role for user that designs course content and structure but Designer who should not have access to actual student data. Grader Subclass of role needed for role of grading assignments but not changing course structure Group Specific set of users in a cohort or assigned to a set of assessments or assignments Instructor Subclass of role for user with complete access to course content and student data Knowledge Subclass of Learning Asset that refers to pieces of information that Chunk makes up larger units. May include video, audio, text, images. Learning Class that rolls up all discrete elements in a course, especially ones that Asset achieve a common learning objective. Name Aids presentation Open Subclass of group needed by opportunity engine Section Distinct roster tied to a course. Student Subclass of role for student taking a course Teaching Subclass of role for user who can do subset of instructor and course Assistant designer User Class to roll up roles and to hold common attributes such as active (y/n)
82 Value Classes Class Name Accommodations Age Assessment Assignment Defined Delivery Type Demographics Gender Hearing Highest_Education Language Learning Dimensions Learning Disability Physical Disability Sight Social Preferences Term Year Description User accommodations for disabilities help determine the electronic presentation of learning assets and courses. Dimension of demographics for analytics engine Subclass of Learning Asset needed to group all 'quiz/test' elements that assess achievement of learning objective Subclass of Learning Asset needed to group all 'turn in' based elements that assess achievement of learning objective Subclass of course type for courses with specific dates for the start and finish. Categorizes courses into types of definitions based on whether the class will proceed on a schedule, wait for critical mass or is open. Set of characteristics about a user that lends to analytics and opportunity suggestions Dimension of demographics for analytics engine User accommodations for disabilities help determine the electronic presentation of learning assets and courses. Dimension of demographics for analytics engine Dimension of demographics for analytics engine, aids presentation Dimension of demographics for analytics engine, aids presentation User accommodations for disabilities help determine the electronic presentation of learning assets and courses. User accommodations for disabilities help determine the electronic presentation of learning assets and courses. User accommodations for disabilities help determine the electronic presentation of learning assets and courses. Dimension of demographics for analytics engine Provides a way to sort and categorize sections tied to academic terms, such as quarter or semester Class for sections and courses to denote year offered of specific instance of the course Service Classes Class Name Collaboration Design Description Subclass of Learning Asset needed to group all synchronous and asynchronous collaboration elements that aid reaching a learning
83 83 Recommendat ions Results Roles objective Set of recommended learning opportunities based on demographics, analytics engine, opportunity engine Suggestions based on analytics engine and learning asset attributes Class to capture specializations of user types to roll up into interface or call as service
84 84 4 Solution Approach 4.1 Architectural Description To build out the operational viewpoint, the Unified Modeling Language (UML) will be used for each of the use cases identified from the stakeholders. The diagrams for each use case were updated in an iterative manner with information learned from additional use cases. This iteration allowed the architecture to evolve as new information was learned Use Case #1 Learning Assets Instructors create and upload a sharable, discoverable learning asset. Figure 19: Use Case #1 Learning Assets
85 Learning Assets Sequence Diagram Instructor adds a learning asset, sends it to the system administrator s queue. The administrator approves and adds the learning asset to the content repository. Concurrent with course creation, the instructor sets initial learning dimensions, such as the asset s dominant learning style(s) and that information is passed to the analytics and opportunity engines. Once the asset is in the content repository, it can be added to a course or learning objective path that then collects analytics of its usage and passes the information to the opportunity engine for recommendations that show on the user s dashboard. Figure 20: Learning Assets Sequence Diagram
86 Learning Assets Class Diagram Learning objectives are goals accomplished by a learning asset or collectively through a course. A learning asset may have several learning goals attached to it, as well as zero to many foundational knowledge requirements. Achievements may be tied to learning assets, learning objectives and foundational knowledge. Figure 21: Learning Assets Class Diagram Course Pre-Decomposition Object Diagram This diagram illustrates the before state of a current face-to-face (F2F) course before it is decomposed into online delivery format as a course, defined learning path or just as a solitary learning asset. A course is made of knowledge chunks, assignments, assessments and collaboration design.
87 87 0..* Lecture (1 HR Video) : Knowledge Chunk 0..* In Class Exercise : Assignment 1 1 AESE 278A :Course 1 0..* Powerpoint : Knowledge Chunk * Classroom Discussion : Collaboration Design 0..* Tutorial Video for CPN :Knowledge Chunk 1 Individual Final : Assessment Figure 22: Course Pre-Decomposition Object Diagram Course Decomposition Activity Diagram This diagram illustrates a generic approach to converting a F2F course by decomposing it first into learning assets (see Learning Asset class diagram) into learning assets that achieve two or less learning outcomes that take less than ten minutes to complete. These numbers could be changed, but should take into account concepts in learning theory such as the optimal amount of information in a learning asset optimized for most efficient retention. For example, the better length might be seven minutes, but could be used a guide. The more discrete a learning asset, fewer learning outcomes it serves, the easier to study (via analytics) and easier to repurpose for many courses and discrete learning paths. This could be carried out by an
88 88 instructor or by a course (instructional) designer. Security is added in future iterative versions of this activity diagram. Figure 23: Course Decomposition Activity Diagram
89 Use Case #2 MOOC Cohorts Students sign up for MOOC cohort groups to facilitate synchronous learning experience. A user can select a cohort group to take a course with. This will connect students to their course, provide camaraderie and connection with the subject matter. Figure 24: Use Case #2 MOOC Cohorts A Cohort is someone who matches criteria such as 'in same area', 'can study at same time', 'shares compatible characteristics for partnership' that one can team with to complete a course. This enables group exercises during the delivery of the course.
90 Cohort Groups State Diagram A user can enhance their experience by joining a cohort group. Figure 25: Cohort Groups State Diagram
91 Cohort Groups Activity Diagram Student joins cohort-based group, signs up for specific group and updates the system. Update triggers either the course to begin or updates the group information. Figure 26: Cohort Groups Activity Diagram
92 Use Case #3 Learning Achievements goal. A student works towards and earns a learning achievement for accomplishing a stated Figure 27: Use Case #3 Learning Achievements
93 Learning Achievements Activity Diagram This activity diagram shows how achievements can stack on each other. Signing up for a single course delivers one Achievement and signing up for a tenth course brings another achievement. Figure 28: Learning Achievements Activity Diagram
94 Learning Achievements Sequence Diagram This sequence diagram illustrates how a student deliberately earns an achievement and the interactions between the MOOC platform, the achievement and learning assets as well as the opportunity engine. Figure 29: Learning Achievements Sequence Diagram Triggered Achievements Sequence Diagram This diagram illustrates that an achievement can also be triggered by an instructor defining a new achievement. The achievement is vetted by the system administrator. The MOOC platform polls the analytics engine for students who have already earned this
95 95 achievement. When the criteria are met, the achievement is registered and the student is notified. The instructor is notified as students reach the achievement. Figure 30: Triggered Achievements Sequence Diagram
96 Use Case #4 Learning Opportunity Suggestions Platform dynamically builds, maintains and pushes learning opportunity suggestions and makes timely recommendations. Figure 31: Use Case #4 Learning Opportunity Suggestions Learning Opportunity Class Diagram Illustrates that the opportunity engine and its relationships. The opportunity engine is largely the recommendations that come from the intersection of user demographics, past courses, etc. In turn the courses taken or learning assets and paths used influence new
97 97 recommendations. The diagram also shows that courses are made of learning assets. Course attributes relate to demographics to influence recommendations for future learning opportunities. Another type of recommendation can be a group to join based on the demographics of the user. This diagram was modified since the previous homework to show the role of the defined learning path, that can be made of several learning assets and several paths may make up a course. Figure 32: Learning Opportunity Class Diagram
98 Use Case #5 Learning Analytics Course designers review analytics from users of learning assets and courses and make pedagogical changes based on that information. Figure 33: Use Case #5 Learning Analytics
99 Learning Objectives Class Diagram Learning objectives are goals accomplished by a learning asset or collectively through a course. A learning asset may have several learning goals attached to it, as well as zero to many foundational knowledge requirements. Achievements may be tied to learning assets, learning objectives and foundational knowledge. Figure 34: Learning Objectives Class Diagram Learning Analytics Sequence Diagram This sequence illustrates and important sub use case Use Case #5, Instructor Creates and Manages Baselines. This illustrates how an instructor modifies a course and then registers a new baseline with the MOOC platform that communicates with the analytics engine. Students
100 100 use the newly modified course and analytics are gathered on the learning assets contained in the course. The instructor can later compare the course analytics before and after the baseline. Such an action might assist the instructor with course assessment and provide quantitative measures of the course or learning asset changes to the course outcomes. The diagram shows the behavior with learning assets organized in a course. The learning assets could also be organized into defined learning paths, likely smaller in scope than a course. Figure 35: Learning Analytics Sequence Diagram Creating a Defined Learning Path Sequence Diagram This diagram illustrates the sequence of behaviors involved in creating a defined learning path. A defined learning path is a set of learning assets presented in a collection that may or may not have a defined linear path, where assets must be seen in a
101 101 particular order. The learning assets have a commonality of achieving a particular learning objective. A rules monitor holds the rules attached to learning paths or courses, such as less than half of the students who completed the path/course achieved the learning objective. If such a rule existed in the monitor, the monitor would mine the analytics continually to see if the rule had been met for notification to the instructor (author) of the rule. The instructor could define a new learning path to facilitate higher rates of learning objective attainment. The diagram shows this as well as the process whereby a system administrator approves the new rule attached to the new learning path. Approval is essential as the rule could be unnecessarily granular and cause system performance issues. At the end of the process, the learning path would be available to users of the systems, both students and for inclusion in other courses. Figure 36: A Defined Learning Path Sequence Diagram
102 A Learning Objective Object Diagram The object diagram shows an example of a learning objective being reached through a learning objective. The student follows a pre-determined learning path consisting of four knowledge chunks. The quiz serves as the assessment piece that measures the learning outcome. Figure 37: A Learning Objective Object Diagram
103 Security Models To address the information technology risks for UCWorld, plans will be created for the following seven areas. Security Policy Management Set uniform security policies Identity Services Provides information about organizational entities users, groups and roles Authentication Services Authorization Services Confidentiality Services Integrity Services Audit Services Application Security Authenticate all user access to system resources. Includes Federated support for students. Apply appropriate access control to all resources Enable data protection for sensitive data. Establish appropriate trust relationships between multiple domains Collect network information and report Deploy secure applications Security Policy Management Figure 38: Security Services Table The MOOC Platform will use a third-party Security Policy Manager to manage locallydeveloped security policies from authoring and publishing to enforcing and updating. The policies will be essential in building IT Services for the MOOC Platform. Failure to identify policies prior to building the system could allow for systems to be configured improperly,
104 104 allowing future security breaches or necessitate rework. Each of the sections below will have their policies stored in the Security Policy Manager Identity Services The MOOC Platform will integrate with a number of different systems, entities and rolls. Given this complexity, the system will be designed using a Single Sign-on (SSO) technology for internal users and a more open federated identity for anonymous students and external users who do not require access to internal systems, official grades or transcripts Social Networking For non-matriculating students or external users, a social networking identity service, such as OpenID (http://openid.net/) or OAuth (http://oauth.net/), will be used where users can create ID s from multiple identity providers such as Microsoft or Google. Figure 39: Open ID Authentication
105 Secure Login For internal users including administration, faculty or students who require official transcripts, system identities will be created that will require different levels of credentials to access systems. An identity matches a Role and Entity as described below. Users will be allowed to have multiple identities such as a Grader from UC San Diego and a Matriculating Student from Harvard Business School if they are both grading a course from UCSD and attending as a student to another program. Entities will include Universities and Organizations. ROLE IT Administrator HR Administrator Administrator Instructor Grader Student (matriculating) Student Observer Teaching Assistant Course Designer SECURITY NEEDS Multi-factor authentication will be required Multi-factor authentication will be required Can only be entered by HR administrator Can only be entered by HR administrator Can only be entered by HR administrator Can be entered by Administrator who checks credentials Uses OpenID or similar Identity Management Uses OpenID or similar Identity Management Can be entered by Administrator or Instructor Can be entered by Administrator or Instructor Figure 40: Roles and Security Needs Application and database level access will include a binding of courses. Students will only be allowed to see courses they are enrolled in, instructors can only add course material and view student work only for courses they are teaching and graders can only grade courses they have been granted permission to.
106 Authentication Services Figure 41: Class Diagram for Security Roles The MOOC Platform will use a SAML 2.0 or similar authentication utilizing Single Signon (SSO). Single Sign On allows users to access all systems without being prompted to log in again at each one of them. Non-matriculating students will be granted access using SAML 2.0 and outside observers to login using an OpenID type federated sign-in to view courses, audit courses and see public facing information. The authentication mechanism will allow access for legitimate users but screen out intrusion attempts. Different levels of access will be utilized from multi-factor authentication for IT and HR personnel, strong password restrictions and routine password updates along with secret questions. The system will be designed loosely coupled so that as new technologies arise that allow us to protect against intruders trying to steal identities or impersonate users, they can be adopted.
107 SAML 2.0 Activity Diagram This activity diagram illustrates how SAML 2.0 or similar technology will be utilized to authenticate a MOOC service. An Agent is a user with a unique login/password [strength of password determined by roles the agent has requested]. The Agent will be established through IT Identity Services. The Service Provider delivers a service to the user that could include viewing data, application access, access to parts of the website. The Identity Provider performs that authentication by checking credentials and ensuring the user is allowed to use the base service. Levels of authorization in the service will be determined by service level roles or application level roles. Figure 42: SAML 2.0 Activity Diagram
108 Authorization Services The MOOC Platform will use the most restrictive access model where the minimum number of network resources, applications and data as possible are given to keep systems secure. Agents will require stronger authentication as they require access to more sensitive data. REQUESTING ROLE ACCESS CONTROL ENFORCEMENT ENTERPRISE RESOURCE AUTHENTICATION ACCESS CONTROL DECISION ACCESS CONTROL POLICY RETRIEVAL ACCESS CONTROL PERMISSIONS RETRIEVAL POLICY MANAGEMENT LABELING MANAGEMENT PERMISSIONS MANAGEMENT SECURITY POLICY REPOSITORY CLASSIFICATION REPOSITORY ROLE PERMISSION REPOSITORY Figure 43: Authorization Services Diagram System level authorization A technology similar to LDAP will be utilized to restrict access to directories and system resources. The roles identified above will be used to partition system access.
109 Application level authorization With each login, a security token will be stored for the user based on their credentials. Each application will use both the token and the roles engine to determine access to screens, data and data entry. The roles engine will determine who can have access to screens within the application and who will receive an error or no data. The application security plans will spotlight how the applications will use the roles engine Confidentiality Services The MOOC will use SSL (Secure Sockets Layer) with asynchronous encryption to ensure confidentiality of information passed from the network to other individuals or service providers. The MOOC platform will be registered with an identity provider, such as Verisign or EnTrust, to secure enterprise level certificates. Both a Public Key Infrastructure (PKI) and Digital Certificates will be employed to ensure secure data is transmitted anytime the data is passed outside the organizational firewall to the Internet. Every time a connection is made to the system, a public key will be returned to the user and will create a session. All data will be encrypted at the user site prior to sending over the Internet. Once received, the servers will decrypt the data with the matching private key. Any data not properly decrypted could be suspected as hacked data.
110 Integrity Services The MOOC platform will use WSDL (Web Services Description Language) in combination with SOAP (Simple Object Access Protocol) for all web service calls. All services that send data in and out of the system will be required to use Security Tokens to identify themselves. Figure 44: Integrity Services - Activity Diagram
111 Audit Services Audit Services analyze individual technical infrastructure components at a detailed level, ensuring that each is functioning in a manner that reinforces appropriate information security Application Security Application Security requires code reviews and training application developers how to write proper code. Code review means searching for vulnerable parts in the program's code itself. Includes looking for vulnerable SQL statements, architecture review, looking for parts that can be called without authorization, points of remote file inclusion, and OS command executions.
112 Executable Models An executable model can be used to model certain aspects of UCWorld as a prototype. It will test certain parts of the system prior to rolling it out to an end user base. For UCWorld, it would be particularly useful to model the formation of cohort groups. By modeling the creation of cohort groups, one can ensure end users will be able to find a group that meets their learning needs Modeling a Cohort Group Because UCWorld will be accessible around the world, students will be accessing the systems from different time zones, speaking different languages and using different learning styles to tackle each class. As students look to join a cohort group, UCWorld will give them the option to select from the following options: Time zone: GMT GMT +8 GMY -8 Language: Mandarin English French Learning Style: Visual Auditory Tactile As different students sign up for classes, they can designate themselves as one of 27 different student types. Examples include: GMT + 8 English Visual GMT French Auditory
113 113 To make sure that UCWorld can properly create different cohort groups, an executable model will be created to generate thousands of student types and statistics on how cohort groups are formed will be collected and graphed. The process to develop the executable model will include coloured petri nets using CPN tools Coloured Petri Nets Coloured Petri nets (CPN Kristensen, Christensen, Jensen, 1998) provide a framework for the construction and analysis of distributed and concurrent systems. A CPN model of a system describes the states which the system may be in and the transitions between these states. UCWorld. To model the creation of cohort groups, a CPN Queue Pattern will be used which ensures strict ordering of objects, while inserting or removing them from the collection. Figure 45: Coloured Petri Net Queue Pattern When the collection of objects in form of a queue is given, and it is necessary to add an object to the collection or select an object from the collection and remove it from the queue. The order in which the objects are being added/removed to/from the collection may depend on
114 114 the properties of an object (e.g., time zone, language, etc.), the location in the queue (FIFO, LIFO), or the timestamp. Student tokens are created using randomizations of the different types. The tokens will be fed in to the queue system with the identifiers and number of students in the cohort group defined. Figure 46: Coloured Petri Net Example The results will show the ability of UCWorld to handle large numbers of students looking to join cohort groups. Average queue times, number of cohort groups created and ideal cohort size can be determined using these prototypes.
115 Service Oriented Architecture UCWorld will be a unified platform communicating with multiple external services. The user will be presented with dashboards and control panels that will allow them to set up options, add classes and view achievements. Each of these services could be happening on different servers with some local and some in the cloud. A class diagram representation of a student interface shows that even the service of adding classes can occur from multiple locations and multiple learning management systems. Figure 47: Student Interface Diagram To enable the interaction between all the services required to create UCWorld and integrate with both legacy applications and new cloud services, a service-oriented architecture will be used. Service orientation is a means for integrating across diverse systems. Each IT resource, whether an application, system, or trading partner, can be accessed as a service. These
116 116 capabilities are available through interfaces; complexity arises when service providers differ in their operating system or communication protocols, resulting in inoperability. Service orientation uses standard protocols and conventional interfaces-usually Web services-to facilitate access to business logic and information among diverse services. Specifically, SOA allows the underlying service capabilities and interfaces to be composed into processes. Each process is itself a service, one that now offers up a new, aggregated capability. Because each new process is exposed through a standardized interface, the underlying implementation of the individual service providers is free to change without impacting how the service is consumed. Figure 48: Service-Oriented Architecture Web Services A web service allows two different services to communicate across the World Wide Web. Current UC San Diego student systems use web services to deliver information about
117 117 students including course information, financial data and personal information. UC San Diego also uses Single Sign On, a web service to deliver identity management information to the students. UCWorld will be designed to handle all external web services including those from UC Office of the President, UC San Diego and any external cloud vendors that are used. An example of using a web service can be seen in how UCWorld will connect to UC San Diego s Single Sign On capabilities. This will allow matriculating students to use a security service that can connect to legacy applications such as Financial Aid, Student Information Systems and Billing Systems with a single ID. Every UC San Diego secure web application requires a login to verify the user and grant them access to the application. The UC San Diego web applications use the campus SSO login and authentication system. Many internal systems are accessed in step 3 in order to authenticate the user. This happens quickly, however if one of those systems is not available, the user is not able to login and access the application. Figure 49: UC San Diego Single-Sign-On Diagram
118 Rich Services The number of requirements defined in this paper goes beyond the ability to simply use web services. Many different systems may be needed including legacy student systems, new analytics engines and external learning management systems. Rich Services (Arrott, Demchak, Ermagan, Farcas, Farcas, Krüger, Menarini, Calit2, UC San Diego) provide a comprehensive and flexible framework for the analysis, design and deployment of large scale and complex web Services based systems. Figure 50: Composed Rich Services Enterprise Service Bus UCWorld will provide an internal enterprise service bus to connect to the larger rich services architecture. Each infrastructure service is connected to the router/interceptor via a service/data connector. Each application service is connected to the messenger. The
UF Online Comprehensive Business Plan 2013-2019 For Consideration by the Board of Governors September 27, 2013 1. EXECUTIVE SUMMARY Overview... 4 Background... 4 Implementation... 5 Timeline Major Milestones...
Online Education Plan For McNeese State University Prepared by: Tom Clay & Associates, Inc. 16 Cedar Lane Cohasset, MA 02025 781-923-1750 firstname.lastname@example.org Prepared for: Dr. Jeanne Daboval Provost
BUILDING A FOUNDATION FOR SUCCESS THROUGH STUDENT SERVICES FOR ONLINE LEARNERS Dr. Ruth Newberry Duquesne University Catherine DeLuca Duquesne University ABSTRACT Retaining students in distance education
Outsourcing Workbook Page 1 Copyright 2008 Notice of rights All rights reserved. No part of this book may be reproduced or transmitted in any form by any means, electronic, mechanical, photocopying, recording,
Priorities for Excellence The Penn State Strategic Plan 2009-10 through 2013-14 Table of Contents The Context for Strategic Planning at Penn State... 1 The Process for Creating the University Strategic
Task Force on Postsecondary Online Education in Florida FINAL REPORT Submitted December 9, 2013 TABLE OF CONTENTS Executive Summary... 1 Trends Relevant to Florida s Online Learning Initiatives... 10 Recommendation
Shaping the Higher Education Cloud An EDUCAUSE and NACUBO White Paper May 2010 EDUCAUSE and NACUBO would like to thank our colleagues at IBM, Gartner, Pearson, and SunGard Higher Education for their generous
E-LEARNING C O N C E P T S, T R E N D S, A P P L I C A T I O N S About Epignosis LLC. All rights reserved. Epignosis LLC 315 Montgomery Street, 8th and 9th Floors San Francisco, California, CA 94104 United
Non-traditional Educational Programs at UM Task Force Report April 2010 1 Table of Contents Overview of Task Force s Work... 3 Key Recommendations... 4 Continuing Education... 8 Space & Partnerships...
Convergence of Social, Mobile and Cloud: 7 Steps to Ensure Success June, 2013 Contents Executive Overview...4 Business Innovation & Transformation...5 Roadmap for Social, Mobile and Cloud Solutions...7
Journal of Information Technology Management ISSN #1042-1319 A Publication of the Association of Management THE APPLICATION OF BUSINESS INTELLIGENCE TO HIGHER EDUCATION: DENNIS GUSTER SAINT CLOUD STATE
MASTER OF PUBLIC ADMINISTRATION You are tomorrow s leader. Prepare today. MPA Southern Utah University Master of Public Administration Student Handbook Revised: August 13, 2014 Southern Utah University
The Definitive Guide tm To Cloud Computing Ch apter 10: Key Steps in Establishing Enterprise Cloud Computing Services... 185 Ali gning Business Drivers with Cloud Services... 187 Un derstanding Business
IOSR Journal of Business and Management (IOSR-JBM) e-issn: 2278-487X, p-issn: 2319-7668. Volume 17, Issue 4.Ver. III (Apr. 2015), PP 32-44 www.iosrjournals.org Succeeding at succession through talent management
CYBERSECURITY WORKFORCE DEVELOPMENT MATRIX RESOURCE GUIDE October 2011 CIO.GOV Workforce Development Matrix Resource Guide 1 Table of Contents Introduction & Purpose... 2 The Workforce Development Matrix
THE THREAT AND OPPORTUNITY OF MASSIVELY OPEN ONLINE COURSES FOR FULL-TIME MBA PROGRAMS Christian Terwiesch email@example.com Karl T. Ulrich firstname.lastname@example.org The Wharton School University
Digital Textbook Playbook The Digital Textbook Collaborative, February 1, 2012 Table of Contents About this Playbook...3 Executive Summary...4 Introduction....6 Making the Transition to Digital Learning...8
Standards for Internal Control in New York State Government October 2007 Thomas P. DiNapoli State Comptroller A MESSAGE FROM STATE COMPTROLLER THOMAS P. DINAPOLI My Fellow Public Servants: For over twenty
Gap Analysis for Information Technology At Sacramento State: A Self Study February, 2008 1 Larry Gilbert Vice President and CIO Information Resources & Technology 2 February 20, 2007 Introduction The California
The Online Master of Distance Education (MDE): Its History and Realization Ulrich Bernath & Eugene Rubin In U. Bernath & E. Rubin (Eds.), Reflections on Teaching and Learning in an Online Master Program.
This Guide to Teaching Online Courses is the product of collaboration among a number of organizations committed to ensuring the quality of online instruction to secondary students in the United States.
JANUARY 2013 REPORT OF THE DEFENSE SCIENCE BOARD TASK FORCE ON Cyber Security and Reliability in a Digital Cloud JANUARY 2013 Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics
University of South Florida Sarasota-Manatee College of Business Strategic Plan October, 2011 1 Executive Summary This strategic plan has been developed for the College of Business (COB) at the University
fs viewpoint www.pwc.com/fsi July 2013 02 14 17 20 27 31 Point of view Competitive intelligence A deeper dive into the benefits of cloud A framework for response How PwC can help Appendix Clouds in the