CANHEIT 2012 Building the Digital University
IT Service Consolidation at the University of Waterloo Bruce Campbell, Director Network Services Martin Timmerman, Director Computing Systems Services Information Systems and Technology
Format of Presentation Bruce Campbell History, Background, Motivation IT Task Force Networks Martin Timmerman AD, e-mail, web Committees, other Next steps
A bit about me Bruce Campbell Director, Network Services, Information Systems and Technology Department, 2007 to present Manager, Science Computing, 2005 to 2007 Engineering Computing (various roles), 1984 to 2005 (Approx 22 years in Faculty IT, 5 years in central IT) I ve seen everything!
University of Waterloo Main campus located in Waterloo Ontario Plus campuses in Cambridge, Kitchener, Stratford, Hunstville, Dubai From http://uwaterloo.ca/aboutuw... Founded in 1957 27,440 full time undergraduate students 3,680 full time graduate students 1,099 full time faculty members 2,184 full time staff members 6 faculties (Applied Health Sciences, Arts, Engineering, Environment, Math, Science) 4 federated and affiliated colleges 10 faculty based schools
IT Organization at UW Central IT department Information Systems and Technology (IST) Significant IT departments in all 6 faculties, and some schools (fully functioning IT departments with data centres, expertise in multiple technologies, etc) IT staff in some departments IT group in Housing recently merged with IST Approx 140 IT staff central out of approx 310 IT staff total.
Partial Org Chart showing central IT department and two faculty IT departments, and University Committee on Information Systems and Technology (UCIST) Dean of Eng Dean of Arts Provost Note: Computing Technology and Services Committee (CTSC) formed 2009 not shown Assoc Dean Computing Assoc Dean Computing UCIST Associate Provost, IST Director, Engineering Computing Manager, Arts Computing
Decentralized IT support at UW Historically (post mainframe era), the central IT group (IST) and largest faculty/department IT groups have each had the mandate, staff, and budget to provide most required IT services to their respective constituencies. Significant budgetary flexibility within faculties. UW has one of the most decentralized IT structures among Canadian universities. Approx 45% IT staff and IT expenditures central.
Group primarily involved in providing service (partial list, pre 2009) Service Central IT Faculty IT Learning Management ERP Telephones Research Support Computer labs E-mail Network Active Directory IT Security Web Desktop rollover Storage, Backups Printing
IT Task Force In November 2008 the Provost convened an IT Task Force to: collect information on current IT practices, and to make suggestions to improve the effectiveness and efficiency of IT services at UW. Trigger event? Why now? Cost of IT highly visible Duplication of effort in management of infrastructure Greater focus on client service desired in many areas
From the memo announcing IT Task Force......Information technology (IT) services at UW are quite decentralized... There are advantages to decentralization. Staff members who are regularly or permanently resident within a unit can provide timely response to local services request and are aware of requirements unique to that unit. There are disadvantages as well. Decentralization can lead to unnecessary duplication of services and suboptimal use of resources. It can also make it difficult to establish overall institutional directions, standards and best practices....
IT Task Force Mandate 1. Identify the collection of UW essential services 2. Identify the most efficient and effective way(s) in which those services can be provided 3. Make recommendations for changes where services are currently being delivered in ways inconsistent with 2.) above
IT Task Force Membership Geoff McBoyle (Chair) Alan George Dennis Huber Ken Coates Tom Coleman Associate Vice President Academic (in 2008) (and former Dean) Associate Provost Information Systems and Technology (and former Dean, Provost) Vice President Finance Dean of Arts Dean of Mathematics
IT Task Force Consultations with campus groups Questionnaire Meetings etc
IT Task Force Report Delivered June 2009 http://uwaterloo.ca/provost/sites/ca.provost/files/uploads/fil es/it-task-force-report-june-2009.pdf Reviewed and endorsed by Executive Council and Deans Council http://uwaterloo.ca/provost/sites/ca.provost/files/uploads/fil es/it-taskforce-memo-june-2009.pdf 18 recommendations Status of implementation at http://ist.uwaterloo.ca/istplans/implementationoftaskforcer ecommendations.pdf
IT Task Force Report Major Recommendations Centralization/Consolidation of a number of services, including: Network management E-mail Active Directory Web Content Management Creation of Computing Technology and Services Committee
Network Management Network Management was centralized January 1, 2011 (Previously, IST had been responsible for approximately 66% of campus network infrastructure funding and management) IST responsible for Architecture and design of network Network operations Funding Specific service provisioning activities (e.g. DNS additions, switch port configuration) may be performed by faculty/departmental IT to support local activities as efficiently as possible. (done with centrally provided tools, generally not cli access to network devices)
Transition of Network Management Several factors contributed to success and relatively straightforward transition... Central funding provided for some network infrastructure since 2006, subject to following campus standards for equipment and some practices. Central network management tool in use since 2005, for switch port configuration etc Central DNS/DHCP management tool Wireless already centrally managed Many practices already standardized (e.g. OSPF) Campus Network Advisory Group (CNAG) in place, to discuss practices, standards, evolution of services, etc. (since renamed Campus Network Services Committee)
Success depends on... Support of senior administration Central funding Good service Commitment to tools which empower faculty/departmental IT staff to provision service and provide first level problem investigation. Commitment to increased self service, pre-provisioning of service. Documented process to augment central services. Clarity. Faculty/department IT staff no longer involved in network management need clear direction from their management, and other work to transition into.
IST Network Services Group Director Senior Technologist / Team Lead NOC 4 network support specialists Team Lead Cable Plant 4 network technicians Telecommunications Services and Physical Security Systems also in group (not affected by centralization of network management)
Policy Minimal formal policy in place. UW Procedure 1 contains: Any contract or agreement which includes purchase of equipment or services with potential to impact UW's IT infrastructure must have prior endorsement of the Associate Provost IST before an agreement or contract can be executed. Network Management documents at: https://strobe.uwaterloo.ca/~twiki/bin/view/istns/networks ervicesresources (includes definition of network management, procedure for augmenting central services, etc)
UW s Network Infrastructure Cisco core, border, VPN HP (Procurve) switches and routers in buildings (approx 1,300 devices) All new (or upgraded) buildings have all wall jacks live, gigabit POE. All new (or upgraded) buildings use VoIP phones Aruba wireless deployed in all campus buildings including residences. (approx 2,000 APs) 2 gbps (soon to be 3 gbps) general external network service 1 gbps (soon to be 10) research external internet service (ORION) Infoblox IPAM Locally developed network management tool (called ONA) Sandvine traffic management for residence and wireless Juniper SRX for machine room firewall Juniper SRX for wireless NAT
Empowering faculty/departmental IT staff to provision service Our network management tool (Open Network Administrator (ona)) allows IT staff to: Find switch port based on IP address/name of host Change switch port settings, including vlan Disable/enable switch port View switch port graphs, statistics, errors, and switch syslog And much more Our IP Address Management tool (Infoblox) allows IT staff to create, change, domain names, DHCP settings, etc.
Next Steps (on network) Major projects under way Security Architecture audit Border firewall Wireless IDS, AUP display/acceptance Campus wide network documentation Disaster Recovery audit Increased redundancy Updates to fibre plant Service initiatives Wireless expansion External network bandwidth upgrades IPv6
A bit about me Martin Timmerman Director, Computing Systems Services, IST Department, 2005 to present Staff and management in Data Processing, Department of Computing Services, and Information Systems and Technology from 1981 to 2005 University of Waterloo for 31+ years, all in central IT
Microsoft Active Directory From the IT Task Force Report Recommendation 2: The University should consolidate, to the maximum extent possible, Active Directory forests and domains, with a preference to move to NEXUS, if feasible. Active Directory Consolidation and Future Governance Merge two main Active Directory domains on campus, ADS and NEXUS Create a campus AD Management Committee for governance/review IST responsible for the single consolidated AD APIST should decide if new AD to be built, or merger into existing Consolidate by merge to NEXUS Size of NEXUS, with more software distribution and higher workstation count Also who needed to do most of the work, keep burden off the Faculty IT groups and assign most of effort to IST staff
Active Directory Consolidation Project effort with campus participation Secondment of key architect from Faculty IT to lead project Design of new OU structure Addition of all users into NEXUS, plus provision from campus identity system Migration of workstations, 21% complete Change of authentication services, first Exchange email, next student facing (email, learning management, student registration, etc.) Introduction of servers (monitoring, database, applications) Change of domain level responsibility, with significant delegation
Campus Email Services Early selected common services forwarding from user@uwaterloo.ca to user@mailhost.uwaterloo.ca spam processing, attachment and content filtering webmail front-end to multiple IMAP servers Pre-task force effort on student email (2006-07) consider shared email service or sharing expertise to deliver a common technology considered enterprise email (Exchange), appliance (Mirapoint), outside service (Google Apps) and open source (sendmail/imap/cyrus) built open source email service and invited campus groups to join two faculties migrated student email to shared service Support staff email (2008) heavy Eudora usage, but moving to Outlook pilot investigation to try Exchange Server primarily campus support staff, but selected participation by faculty IT
Email Recommendations From the IT Task Force report: Recommendation 1: The provision and management of the technical infrastructure for the services listed above should become the responsibility of IST after consultation with the relevant local client group(s). Referred specifically to email services for students Further, IST could be responsible for delivering primary email service to all University faculty and staff
Email Services for Students Implementation phase build up of infrastructure, including additional storage, automatic provisioning focused first on incoming class of September 2010 existing mechanisms and support structures difficult to change then moved upper year students over the Fall 2010, Winter 2011 terms original two faculties have actually retired email servers, other four still have servers in operation, although mostly no student email load System statistics 69,000 accounts 3.2 TB of email data spread over 4 mailbox servers 45,000,000 messages Dell servers, Redhat Enterprise Linux, Cyrus Email, some SquirrelMail usage, with Horde framework outside the cluster
Exchange Server Exchange Server Implementation Pilot implementation in 2008-09, supporting only email with Exchange Server 2007 Calendar function provided by Oracle Calendar BlackBerry Enterprise Server (BES) for significant usage at Waterloo Early 2010 investigation to migrate away from Oracle Calendar to Exchange Calendar, but wait for Exchange 2010 version Upgrade to Exchange Server 2010 in Fall 2010 Migrate to Exchange Calendar on a weekend in December 2010 Provided many in the faculty supported community with calendar only accounts With integrated email/calendar now possible, focus to migrate employees to Exchange Underway, but slow progress Variety of email clients, heavy reliance on client visits to migrate
Usage of Exchange Server 2.5 years of growth Current usage
Campus Email Services Committee Communication forum to discuss the services Promotion of tools to support users Plan for potential changes, including the authentication change Enhancements such as sendit, a service to distribute large files outside of email Started in Winter 2012
University Web Service From the IT Task Force: Maintaining a state-of-the-art web site for the University involves the coordination of content, design and technical infrastructure. The provision and management of the technical infrastructure properly belongs with IST; the design and content development of the web space belongs elsewhere within the University. Existing campus bodies: Web Advisory Committee and Web Steering Committee Campus project to choose a CMS (twice) Project to implement Drupal for Web Content Management System (WCMS) One area in IST with significant new staffing
New WCMS Service Web resources site at http://uwaterloo.ca/web-resources/ Hands on planning sessions, with migration assistance by coop students hired for the purpose Four faculty level sites and 70 sites in total Not UW home page yet, coming in Fall
New IT Committee - CTSC Computing Technology and Services Committee Director level management with staff resources IST, Faculty IT, Library, Housing Encouraged to be university officers considering campus as a whole and not individual campus unit Regular forum for discussion Creation of CTSC projects and working groups (Microsoft licensing, Green IT, campus printing, AD Consolidation)
Next Steps Complete the initiatives (remove email servers, remove ADS, remove web servers) Digital Asset Management project underway Campus shared file service Mobile applications for the campus Helpdesk coordination, investigation of common toolset Mindset is starting to change, but old habits hard to change New CIO starts in July