Los Angeles Unified School District Office of the Inspector General Internal Audit. Audit Report. Network Infrastructure and Equipment Upgrades

Transcription

1 Los Angeles Unified School District Office of the Inspector General Internal Audit Audit Report Network Infrastructure and Equipment Upgrades OA December 26, 2012

2

3 About the Office of the Inspector General The Office of the Inspector General reports directly to the Board of Education. We conduct independent audits, reviews and investigations of District operations, contracts and vendors in order to: Find ways to improve processes, programs, functions and activities Provide information that supports effective decision making Identify real or potential misuse of District resources Prevent and detect waste, fraud and abuse within the District Through our work, we strive to encourage a culture of accountability, transparency, collaboration and excellence and to assist the Board and the Superintendent in their efforts to provide a high quality education for the students and parents of the Los Angeles Unified School District.

4 Why did the OIG do this audit? Advancements in technology are increasing the options available to teachers to deliver instructional initiatives as well as changing the way the students learn and perform. On December 13, 2011, the Board of Education approved bond funding of the local area network (LAN) modernization projects for 138 specific schools. These schools have aging IT network infrastructure and obsolete equipment which limits our students' access to, and use of, electronic instructional initiatives and the internet. The plan is to first update the LAN at these schools, and once the upgrades at all 138 schools are complete, every existing LAUSD K-12 school will be brought up to a minimum standard that includes an up-to-date LAN, campus-wide wireless and access to electronic instructional initiatives. What was the main objective? The objectives of the audit were to (i) determine the reasons the Local Area Network (LAN) Modernization Project budgeted cost per school is more than the cost of the short-term solution implemented at Chatsworth Park Elementary School and compare the scope of work and resulting capabilities of the system, (ii) determine how the LAN Modernization budgets for the schools were developed, and (iii) determine if the LAN Modernization contracts were procured competitively. What OIG goals does this audit support? Provide information that supports effective decision making. Identify real or potential misuse of District resources. Prevent and detect waste, fraud and abuse within the District. What District key strategies does this audit support? Transform teaching and learning so that we prepare all youth to graduate college and workforce ready. Ensure a safe, caring, and nurturing environment for all youth. Operate an effective, efficient, and transparent organization in order to assure the public trust. Areas requiring attention and what the District should do next are discussed briefly in the Executive Summary of this report.

5 TABLE OF CONTENTS Page Executive Summary 1 Background 5 Discussion 7 Exhibit A Photographs of LAN modernization work 13 Exhibit B Photograph of equipment installed as a short-term solution 19 Annex A Response from ITD on audit recommendation 20 Annex B Distribution list 21 Annex C Audit team 22

6 EXECUTIVE SUMMARY Introduction This report contains the results of our audit of the Network Infrastructure and Equipment Upgrades. We performed this audit as part of our FY 2013 Work Plan. The Information Technology Division (ITD) is responsible for building and maintaining the information technology environment needed to support 21st century teaching and learning in the Los Angeles Unified School District. Objectives The objectives of the audit were to (i) determine the reasons the Local Area Network (LAN) Modernization Project budgeted cost per school is more than the cost of the short-term solution implemented at Chatsworth Park Elementary School and compare the scope of work and resulting capabilities of the system, (ii) determine how the LAN Modernization budgets for the schools were developed, and (iii) determine if the LAN Modernization contracts were procured competitively. Summary of Key Audit Findings 1. Why do the LAN Modernization Projects cost so much more than the short-term solution implemented at Chatsworth Park Elementary School? What is the difference between the scope of work and the capabilities of the resulting system? The District has the vision of creating a Virtual Learning Complex (VLC), a 21st century learning environment at LAUSD schools that supports a computing device for every student, facilitating learning opportunities anytime or anywhere. This environment requires the proper LAN infrastructure and equipment at LAUSD schools in order to allow every student and teacher to wirelessly access the internet for state-of-the-art instruction and global access to information. The LAN Modernization Project provides this necessary infrastructure and equipment. The scope of work is very broad and includes redesigning of the local area network (LAN)/wireless local area network (WLAN), upgrading/installing equipment, replacing/installing cables, and constructing pathways. The resulting LAN/WLAN is a robust state-of-the art system that provides fast and reliable connection to 100% of students/teachers campus-wide, and is anticipated to last at least seven years. In contrast, the short-term solution implemented at Chatsworth Park Elementary School is much smaller in scope and involved only the connection of 11 wireless access points to existing equipment. Though less costly, the resulting system allows only a limited number of users to 1

7 simultaneously access the network wirelessly, provides a less rapid and reliable connection, and in addition, it is expected to last for only two years. 2. How were the LAN modernization project budgets developed? ITD indicated that its LAN modernization project budget for each school was estimated based on (i) historical data, or (ii) the contract amount plus 20% for District overhead cost if the LAN modernization contract amount was available. We tested ITD s LAN modernization project budget amounts by computing the estimated project costs for four LAUSD schools (based on the schools LAN modernization contract amounts 1 plus 20% for District overhead costs) and compared the computed estimates with ITD s project budgets for the schools. We found that the budgeted amounts were within 5% of the estimated project costs for three of the four schools. However, the LAN modernization budget for Nobel Middle School exceeded the estimated cost by approximately $832,000 or 52%. ITD indicated that the project budget for Nobel Middle School was estimated based on historical data before the contract amounts were available, and that it would revise the LAN modernization project budget for Nobel MS in its next Strategic Execution Plan (SEP). Recommendation ITD should update its LAN modernization project budgets as the latest contract amounts and overhead costs become available. ITD should also make the appropriate revisions to the project budgets presented in the annual ITD Strategic Execution Plan (SEP) to ensure that the SEP contains the most current and accurate information. This will assist ITD management in their strategic planning for the use of the allocated bond funds and continuing transparency to the Board and other stakeholders. ITD s Response ITD agreed with the recommendation and indicated that it would update its LAN modernization project budgets as the latest contract amounts and overhead costs become available and would also make the budget revisions in its next SEP. Inspector General s Response ITD s response to our audit finding and recommendation indicate an understanding of the issue presented. We consider ITD s planned actions to be constructive steps improving the processes reviewed in this audit. 1 Contract No , School Summary, Package 3, Page 81 2

8 3. Were the contracts with telecommunication firms to perform LAN modernization work procured competitively? Procurement Services Division (PSD) procured the LAN modernization contracts competitively using the best value procurement method. The bidders were scored based on experience, past performance, qualification of personnel, solution requirements, and price. The highest scoring responsible bidder is awarded the contract. Conclusion: Based on our audit, we conclude that the additional costs of the LAN modernization projects over short-term solutions are justified because: i. The LAN modernization projects scope of work is significantly broader than those in short-term solutions. ii. iii. The majority of the LAN modernization costs are paid to competitively-procured contractors to perform the work. Unlike short-term solutions, the LAN modernization projects result in systems that provide high density wireless environments (to support an unlimited number of simultaneous users) and high capacity bandwidth WiFi networks (to allow students to stream videos simultaneously from network servers on the premises) that is capable of supporting an Empowered Learning experiences for 21 st Century learners. Scope and Methodology This audit covered LAN modernization projects for the 138 schools approved by the Board of Education for additional bond funding on December 13, We performed the audit from September 7, 2012 to October 9, We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In order to address the audit objectives, we performed certain procedures, which included, but were not limited to the following: Interviewed ITD staff responsible for the LAN modernization implementation. Reviewed District policies and procedures as they related to LAN modernization contract procurement. Reviewed the LAN modernization contract procurement package (RFP-1082) for documentation of competitive bidding, bidders evaluation and award. 3

9 Visited schools to interview ITD staff, observed work in progress, and took photographs of LAN modernization work and a short term solution. Validated the project budget of four schools against supporting documentation. 4

10 BACKGROUND The Los Angeles School District s (District) vision is that every student will receive a quality education in a safe, caring environment, and will be college prepared and career ready. One of the District s key strategies is to transform teaching and learning so that all youth graduate college-prepared and workforce-ready. 2 ITD s vision, in support of the District s vision and strategy, is to provide District schools optimized information and communication infrastructure that supports a high performance learning environment through the 21 st century. To realize this vision, ITD framed and began work on an initiative called the Virtual Learning Complex (VLC). The Virtual Learning Complex Program The VLC is a holistic approach to learning, centered on four stakeholder groups: students, their families, teacher/principals, and central/local administrators. The goal of the VLC is to deliver schools a modern network environment that is capable of supporting an Empowered Learning experiences for 21 st Century learners. Parents and families can access the VLC to engage and collaborate with teachers and to monitor and motivate their learners. Educators can access the VLC to view student profiles, gain administrative support, create lesson plans and use as a professional development tool. Finally, administrators (central and local) can access the VLC for meaningful, timely data to assess and track students, to provide instructional and teaching support, to perform operational and administrative tasks and to provide staff development. 3 LAN Modernization Project Building a technology environment that supports 21 st century instructional initiatives requires the infrastructure of every school site to be VLC certified. This means meeting two key criteria: 1. High capacity bandwidth: modern wiring that allows high volumes of data to flow in and out. 2. High-density wireless: a system that enables a virtually unlimited number of student, teachers, and other to access the network simultaneously. Today a very limited number of District K-12 schools have Local Area Network (LAN) infrastructures that will support VLC standards. The vast majority of LAN projects undertaken over the past decade occurred prior to the introduction of the high throughput and wireless technologies that are used today, and upon which current instructional technologies depend. On December 13, 2011, the Board of Education approved bond funding of the LAN modernization projects at 138 schools that have aging IT network infrastructure and obsolete equipment, limiting students' access to, and use of, electronic instructional initiatives and the 2 LAUSD s Strategic Plan Information Technology Strategic Execution Plan April 2012, Pages 9 5

11 internet. The plan is to first update the LAN at these schools, and once the upgrades at all 138 schools are complete, every existing LAUSD K-12 school will be brought up to a minimum standard that includes an up-to-date LAN, campus-wide wireless and access to electronic instructional initiatives. ITD estimated the project cost to range from $600,000 for a small school and $2.8 million for a large school (see Table 3 of this report). In June 2012, the District contracted with a telecommunications company for $10.8 million for modernizing the LAN at 14 of the 138 schools. Short-Term Solution ITD installed equipment at Chatsworth Park Elementary School in the fall of 2011 to provide immediate wireless access to the internet upon request from the school. It was meant to be a short-term solution to provide wireless access before the LAN modernization project could be completed at the school. The short-term solution cost the school about $12,000, which was less than 2% of the $740,000 budgeted cost for LAN modernization at the same school. 6

12 DISCUSSION 1. Why do the LAN Modernization Projects cost so much more than the short-term solution implemented at Chatsworth Park Elementary School? What is the difference between the scope of work and the capabilities of the resulting system? ITD has the vision of creating a Virtual Learning Complex (VLC), a 21st century learning environment at LAUSD schools that supports a computing device for every student, facilitating learning opportunities anytime or anywhere. This environment requires the proper LAN infrastructure and equipment at LAUSD schools in order to allow every student and teacher to wirelessly access the internet for state-of-the-art instruction and global access to information. The LAN Modernization Project provides this necessary infrastructure and equipment that allows unlimited campus-wide wireless access to the network and supports the use of classroom computing devices (such as tablets computers, smart phones and boards, ipad, ipad2, Kindle, Nook, Touch Pads, laptops, netbooks, projectors, printers/scanners, and other communicative devices). The scope of work is very broad and includes: Redesigning of the local area network (LAN)/wireless local area network (WLAN) leveraging existing equipment Upgrading/installing equipment Replacing/installing cables and conduits Constructing pathways for laying cable The resulting LAN/WLAN is a robust state-of-the art system that provides reliable connection to all students and teachers campus-wide. ITD estimates that the technology will last for over seven years. See Exhibit A for pictures of LAN modernization work completed at Ambler Avenue Elementary School. ITD has also implemented short-term solutions at some schools at reduced costs. These short-term solutions scope of work was significantly less than that in the LAN modernization projects. For instance, the short-term solution implemented at Chatsworth Park Elementary School involved only the connection of 11 wireless access points to existing equipment to transmit and receive WLAN radio signals. Exhibit B shows one of such access points installed. According to ITD, the short term solution results in a relatively less reliable system that can accommodate a limited number of users at a time connecting at slower speeds, and will last for up to two years only because of older technology equipment being used. ITD indicated that the short-term solution was provided to the school to allow the school to have immediate wireless accessibility. 7

13 Table 1 provides a comparison of the LAN modernization project versus the short term solution implemented at Chatsworth Park Elementary School in terms of cost, scope of work and capabilities of the resulting system. Table 1 Comparison of the Cost, Scope of Work, and Characteristics of Resulting Systems COST SCOPE OF WORK Are design and engineering involved in creating the LAN/WLAN infrastructure? How many wireless access points are installed? Are access points mounted securely to structures? Are latest technology access points installed? LAN Modernization The budgeted cost for a small school ranged from $624,105 to $748,926. Yes. ITD lays out the specifications in the LAN modernization contract and the contractor designs to meet the specifications while taking into consideration the existing equipment and requirements at each site. Typically access points are installed throughout the campus, indoors and outdoors. See pictures #5,6, 8, 10 and 16 in Exhibit A. Yes. Access points are securely mounted on walls and ceilings with mounting brackets. Wireless access points installed outdoors are enclosed in access point enclosures. See pictures #5,6,8,10 and 16 in Exhibit A. Yes. IEEE n standard wireless access points are installed. Wireless-N networking standard offers faster and more stable connection than wireless-g standard. Short-Term Solution $12,000 No. The access points were connected to existing equipment. 11 access points were installed indoors only. No. Access points are places on tables and not mounted to structures. See picture #12 in Exhibit A. No. IEEE g standard wireless access points were installed. Is the core switch replaced? Yes. See pictures #1 and 2 in Exhibit A for the new and old core switches. No 8

14 LAN Modernization Short-Term Solution Are new conduits installed? Yes. See picture #7 in No Exhibit A. Are cables replaced? Yes No Is the network reconfigured? Yes No Is this a construction project with possible Yes. See picture #9 for site No trenching, paving, painting, asbestos removal, electrical and air conditioning work? where trenching work was performed. Is the work covered by Project Stabilization Yes No Agreement? Is quality assurance and site inspection Yes No required by District staff? What is the funding for the project? Bond Funds General Funds CHARACTERISTICS/CAPABILITIES OF RESULTING SYSTEM How many users can access the network wirelessly simultaneously? An unlimited number of users can access the network simultaneously. Only a limited number of users can access the network simultaneously. Where can users access the network? Users can access the network campus-wide. Users can access the network only at limited areas near the 11 installed access points. How is the connection speed? Fast connection speed because of the high network bandwidth. Slower speed. connection Is the wireless signal reliable? Very reliable signal. Less reliable signal. Radio frequency interference could cause spotty coverage and dropped connections. Does the installed equipment meet ITD Yes Yes specifications? Does the installation meet ITD specifications? Yes No Is the bandwidth scalable according to need? Yes No How long will the system last? Over 7 years. Up to 2 years. 9

15 2. How were the LAN modernization project budgets developed? The cost of a LAN modernization project includes costs related to: (i) design and engineering, (ii) installation and configuration of network equipment, (iii) cabling and cabinetry, and (iv) project management, quality assurance and inspection performed by the District. According to ITD, factors impacting the cost of the LAN modernization project include the size of the school, the school site conditions and needs, age of existing network equipment, layout of buildings, the need for trenching, condition and extent of existing pathways and cabling, configuration of classrooms and the distance between classrooms. Table 2 shows ITD s estimated cost of the LAN modernization projects for schools of different sizes. 4 Table 2 Typical Cost of LAN Modernization Projects by Size of the School School Size Estimated Cost Small $600,000 to $749,500 Medium $900,000 to $1,728,000 Large $2 million to $2.8 million ITD indicated that its LAN modernization project budget for each school was estimated based on (i) historical data, or (ii) the contract amount plus 20% for District overhead cost if the LAN modernization contract amount was available. Overhead costs consist of cost associated with procuring the contract, administering the contract, and overseeing the contractor s work (including reviewing/approving the design and inspecting the completed work). The contract amount for each school depends on various factors such as the size of the school, the school site conditions and needs, age of networks, layout of buildings, the need for trenching, condition and extent of existing pathways and cabling, configuration of classrooms and the distance between classrooms. We tested ITD s LAN modernization project budget amounts by computing the estimated project costs for four LAUSD schools (based on the schools LAN modernization contract amounts 5 plus 20% for District overhead costs) and compared the computed estimates with ITD s project budgets for the schools. We found that the budget amounts were within 5% of the estimated project costs for three of the four schools. However, the LAN modernization budget for Nobel Middle School exceeded the estimated cost by approximately $832,000 or 52%. Table 3 shows the comparison. 4 Information Technology Strategic Execution Plan April 2012, Appendix V, Priority Schools for VLC School Network Modernization, Page V-1 5 Contract No , School Summary, Package 3, Page 81 10

16 School Name Chatswort h Park ES Table 3 Comparison of Project Budgets with Computed Estimates School Size LAN Modernization Contract Amount (A) 20% District Overhead Cost (B) Estimated Project Cost (C) = (A) + (B) Project Budget per the SEP (D) Difference (E) = (D) - (C) % Difference (F) = (E)/(C) Small $624,105 $124,821 $748,926 $740,488 ($8,438) -1% Hale MS Medium $1,344,701 $268,940 $1,613,641 $1,691,707 $78,066 5% Woodland Hills Academy Large $885,466 $177,093 $1,062,559 $1,044,000 (18,559) -1% Nobel MS Medium $1,335,064 $267,013 $1,602,077 $2,434,390 $832,313 52% According to ITD, the project budgets for the above four schools were estimated based on historical data before the contract amounts were available. ITD also indicated that it would revise the LAN modernization project budget for Nobel MS in its next Strategic Execution Plan (SEP). Recommendation ITD should update its LAN modernization project budgets as the latest contract amounts and overhead costs become available. ITD should also make the appropriate revisions to the project budgets presented in the annual ITD Strategic Execution Plan (SEP) to ensure that the SEP contains the most current and accurate information. This will assist ITD management in their strategic planning for the use of the allocated bond funds and continuing transparency to the Board and other stakeholders. ITD s Response ITD agreed with the recommendation and indicated that it would update its LAN modernization project budgets as the latest contract amounts and overhead costs become available and would also make the budget revisions in its next SEP. 3. Are the contracts with telecommunication firms to perform LAN modernization work procured competitively? To determine the procurement method of the LAN modernization contracts, we interviewed ITD and Procurement Services staff and examined documentation associated with the procurement of Contract No including the Request for Proposal (RFP-1082), the advertisement for the RFP, the bids submitted by telecommunication firms in response to the RFP, the source selection committee scoring matrix and the procurement summary. 11

17 The contract was competitively procured using the best value method. Procurement Services Division (PSD) publicly advertised the RFP; seven firms submitted proposals in response to the RFP; the source selection committee scored the proposals based on the various criteria (experience, past performing, qualification of personnel, solution requirements, and price); and PSD awarded the contract to the highest scoring responsible firm. 12

18 Photographs of LAN Modernization Work EXHIBIT A Picture #1 - The new core switch. Picture #2 - The old core switch that was changed out. 13

19 Photographs of LAN Modernization Work EXHIBIT A Picture #3 - This is an uninterruptible power supply (Tripp Lite UPS SU6000RT3U). Picture #4 - The top unit is the old Cisco 3600 Series router. The old router was removed as part of the network equipment migration. The bottom unit is the new Cisco 3945 router. 14

20 Photographs of LAN Modernization Work EXHIBIT A Picture #5 A wireless access point installed on the ceiling in a classroom. Picture #6 A wireless access point installed on the wall in the library. 15

21 Photographs of LAN Modernization Work EXHIBIT A Picture #7 - Conduit and cabling installed. Picture #8 A wireless access point mounted on a wall facing the basketball court. 16

22 Photographs of LAN Modernization Work EXHIBIT A Picture #9 - Site where trenching work was performed. Picture #10 - A wireless access point mounted on the multipurpose room outside wall. 17

23 Photographs of LAN Modernization Work EXHIBIT A Picture #11 - Wireless access points installed along the walls in the multipurpose room. 18

24 Photograph of Equipment Installed As A Short-Term Solution EXHIBIT B Picture # 12 This is one of the 11 wireless access points that was installed at Chatsworth Park Elementary School as a short-term solution. (Picture provided by the School s Principal) 19

25 ANNEX A 20

26 ANNEX B DISTRIBUTION LIST Members, Board of Education Superintendent of Schools Senior Deputy Superintendent of School Operations General Counsel 21

27 ANNEX C AUDIT TEAM Stella Lai, Audit Manager Alma Gonzalez, Principal Auditor 22

28 Know about fraud, waste or abuse? Tell us about it. Maybe you are a School District Call the Hotline: Employee, or maybe you are a private citizen. Either way, you are a taxpayer. (213) Or LAUSD-OIG Maybe you know something about fraud, or waste, or some other type of abuse in the School District. Write to us: The Office of the Inspector General has a Fraud Hotline Center hotline for you to call. You can also write 333 S. Beaudry Ave., 12th Floor to us. Los Angeles, CA If you wish, we will keep your identity confidential. You can remain anonymous, if you prefer. And you are protected by law from reprisal by your employer. Website: 23