Phase Two Report Dear Client Regarding the XXX Damage case SUMMARY This report; titled the Phase Two Report, (phase one was the final report August 2013) provides and analysis of the practicality and probable cost impacts of alternative courses of action XXX could have chosen to follow for the replacement / repair of their security network. XXX has indicated that equipment such as controllers / switches (Core Network Elements) sustained water damage to their motherboards. This statement appears to have been offered as explanation for the decision to replace the entire network rather than a possible alternative course of action suggested below. The alternative course of action, being an alternative equipment replacement / repair scenario, was framed as questions in communications with your representative. 1. Would it have been more time efficient and cost effective to remove and replace the controllers and switches only? 2. Would it have been more time efficient and cost effective to remove and replace the motherboards only? 3. Was the course of action XXX chose a reasonable choice from a technology perspective? 4. Additionally, I was asked to provide an analysis of the potential salvage values of both the damaged and undamaged equipment. To save the reader time, the direct answers to the above are: 1. No. Circumstances and the urgency of the situation make this alternative impractical. 2. No. This is not a cost effective option. 3. Yes. Given the high impact risk of running a casino without video surveillance, it is understandable that the decision was made to remove all equipment as quickly as possible and replace the entire system. As it pertains to electronic equipment, replacement is always faster and frequently less expensive than repair. Page 1
4. The damaged equipment has zero salvage value. The undamaged equipment has a low salvage value for two primary reasons; it is outdated equipment and it has not been determined specifically which equipment is damaged or undamaged, making it a high risk purchase from a buyer s perspective. A full explanation of the process, rationale and supporting evidence used to arrive at these answers is provided in the body of the document. PROCESS Print and CD Review I reviewed both the printed pictures and the CD provided by XXX for evidence of specific types of damaged equipment, or pictures of opened equipment documenting damaged motherboards. All of the pictures are more general, providing views of equipment racks, stacked equipment, ceiling damage, wet floors and workers trying to contain the water and/or removing equipment. Some of the printed pictures show water damaged equipment, but because it was necessary to take the pictures in close up to show the water spots, it is impossible to identify the type of equipment. None of the pictures are of opened equipment. The video on the CD captures what seems like a few minutes of the activity during or immediately after the event. The picture and video evidence is of no direct value in validating which equipment was wet or the extent of the damage. The most reliable source for that information remains the visual inspections performed last August. Review Inventory of Damaged Equipment Last August; a complete inventory of all equipment that XXX declared damaged, was performed by XXX s technical expert, a claims adjustor representing the insurance company and myself. I was hired by the legal firm representing the contractor. Together, the three of us examined approximately 900 pieces of equipment. The examination was limited to a visual inspection of the outside of each device. We inspected for the appearance of water spots and/or water-borne sheetrock particles. The presence of undisturbed, accumulated dust on the equipment indicated the device was never wet, and possibly not present in the security room at the time of the event. There was a notable minority of equipment that did not appear to have been in the security room at the time of the event. This inventory is still the most reliable assessment of damaged and undamaged equipment. The inventory has been data sorted and saved so that it may be identified by equipment type and/or manufacturer. It may also be traced back to the pallet it was Page 2
stored on. Each entry in the file includes noted observations as to water spotting and dirt residue, or lack thereof. These files are used for the purposes of this analysis to identify equipment types that most likely have motherboards in their circuit architecture. Motherboards are common to the circuit architecture of multi-functional, multi-process equipment such as computers, switches and network controllers. The devices identified in the following table, derived from the aforementioned inventory, are most likely to have motherboards. As one might anticipate, with the exception of the Dell computers, these device are the more expensive elements of the network comprising a major percentage of the financial loss. Manufacturer Description Model Vicon Alarm Monitor V1300X-IA-EXP Vicon Communications Distribution Unit 20000 Vicon CPU Alarm Monitor V1400X-CCM Vicon Digital Mux AURC-CD500 Vicon Pilot 66/99 Card Cage V6680SCC-MG Vicon Time Sync TL158A-R4 Vicon Video Controller V1400X-VLD Vicon Video Switch V1300X-TCC Vicon Video Switch AUR2K-DC6 Dell Computer Dell Dell Computer Dell Dell Computer Dell Pelco Video Switch VA6220 Internet Research I searched on the Internet for product descriptions and (ideally) schematic diagrams of the equipment. The Internet is frequently an archive of older information as well as a resource for up to the minute information. My thought was that any product information once posted at some location on the Internet may still be there even though the equipment is no longer current model or even if the manufacturer is out of business. I was successful finding product specification documents for most of the devices in the table, but did not find any schematic diagrams. The product literature does not tell me with certainty about the circuit architecture of the device. I will use the simple logic that the more complex equipment, that performs switching and control processes are the most appropriate equipment pieces relevant to the purposes of this Page 3
analysis. I chose to cast as wide a net as reasonable in choosing which equipment is applicable because it is not clear what specific equipment XXX may have been referring to when they stated the equipment had to be replaced because the motherboard had been damaged. I understood that statement to mean that any equipment with a damaged motherboard is a total loss, an assumption with which I completely agree. EXPERT OPINION Considerations Affecting Repair / Replace Decision Specific Equipment Replacement (by the box) One value of the pictures and video is they establish the volume of water, location of the leak in the ceiling directly above equipment racks and the extent of the water spread. The simplest description of the event is that there was water going everywhere. Everything was getting wet to some degree. I think it is also relevant to consider the nature of the work the equipment was performing. The essential purpose of the equipment was video surveillance for the security of guests and most significantly the millions of dollars of betting activity going on 24 X 7 X 365. Given the high impact risk of running a casino without video surveillance, it is understandable that the decision was made to remove all equipment as quickly as possible and replace the entire system. From what I saw during the assessment last August, the equipment was removed by cutting off cables and pulling out entire racks. Given what I am sure XXX views as the overwhelming importance of their security operations, a more surgical approach to sorting out the damaged from the undamaged and replacing only what was necessary was regarded as far too time consuming and risky to the overall casino operations. Replacement of specific pieces; impractical at the time of the event, is still impractical after the event. The reason being the legacy system and equipment was primarily older analog technology and the equipment purchased is modern digital technology. While it is possible to use both technologies in the same system, it would not have been possible at the time to know what could be reused in designing such a network. Purchasing a complete new system was the only practical approach to restoring security operations as quickly as possible. Motherboard Replacement The motherboard is the center of activity within the device typically performing all nonspecialized functions and processes such as circuit interconnection, power distribution and timing. They are frequently the most expensive circuit board in the device because Page 4
they include microprocessors and other typically expensive parts. Equipment that is still in manufacture life-cycle (not discontinued) may well have replacement motherboards available. Equipment that is no longer in manufacture will not have spares available from the manufacturer, but may still be available through after-market resellers or sources such as EBay and Amazon. The question of replacing the motherboard is multifaceted. Not only is there an issue of availability, but technology has made board-level replacement difficult except for a technical repair shop set up to perform this type of work at a large volume (issue of profitability). Regardless of the level of complexity of replacing the motherboard, it is reasonable to understand that XXX would not have equipped a repair shop for their engineers to perform such work as part of their normal processes. In other words, there was no advance preparation that would enable XXX to pursue this alternative efficiently. The reality of the electronics equipment industry, with few exceptions, is that equipment is cheaper and faster to replace than repair. Further to the point, XXX could not have left their security system disabled while they performed the necessary diagnostics, identified the sources for replacement boards, ordered and received the boards, performed the replacement work and then reinstalled the equipment to bring the system back on line. Motherboard replacement as a step in the process to restoring the security system to full operation would have been a grossly impractical course of action. The motherboard replacement question has a second part, that is could the motherboards be replaced, post system restoration. Not all, but many of the devices identified in the table above are manufacture discontinued. Finding replacement boards will be difficult if not impossible. A suitable contractor will have to be identified to obtain the boards and perform the work. Given the limited supply and specialized nature of the equipment it is reasonable to believe the cost would exceed the current value of the equipment. The process would spend dollars to recoup nickels. Salvage Values The assessment made last August included documenting the percentage of water spotting by equipment type, which was 63% of the equipment in the above table. Damaged Equipment There is no salvage market for damaged electronic equipment. Even though there is copper in the wiring and possibly gold plating on the switch contacts, the cost of extraction exceeds the value of the metal. This industry does not equate to the auto industry in which even a car in a head-on collision has reusable parts to harvest. Cars Page 5
are expensive compared to most electronic equipment, so it is still cost effective to repair them, hence a used parts market thrives. Undamaged Equipment The salvage value of the undamaged equipment was indirectly determined last August. In my final report, for the purposes of estimating a loss value, I provided the current value of all equipment. I was not given any guidelines as to the approach the insurance carrier wanted to use in estimating the loss value (e.g. original cost, current cost). Since most of the equipment is older vintage and/or manufacture discontinued, the only source for current pricing information was EBay, Amazon and aftermarket resellers. What I attempted to do was develop the depreciated value of the equipment, using available information and resources to determine what it would cost at that time to purchase the identical equipment. The purpose was not to suggest it as a practical technical solution. It would not have been possible to purchase everything needed to restore the network. It was simply the best means I had to provide an analysis of loss value other than using original cost. The loss values estimated in the report of last August should not be understood as a statement of what XXX should have or could have spent to restore the system. As stated in other sections of this document, piece-part replacement would not have been a technically practical, time efficient or cost effective approach to restoring the network. In the final report I provided the following (at the time referred to as replacement) cost estimates. While the purpose at the time was not for a salvage value estimate, I believe the values are equally valid for that purpose now. Replacement cost of all suspect pieces $374,947 Replacement cost assumption of 25% damaged $93,737 Replacement Cost Most Likely Damaged $202,742 A suspect piece is one showing any water spots. Explanation of this table, copied from the final report of last August is quoted immediately below (Italicized, shaded). My personal and subjective opinion, derived from observations made during the inventory is that the majority of pieces indicated as suspect had such little evidence of water spots that it is unlikely they are damaged. My estimate, based on my observations during the survey, is that no more than 25% of the suspect pieces had a significant amount of water spotting and sheet rock particles in places on the housing that may have allowed infiltration. More important than raw percentages is which products had the significant water spots because some products are considerably more Page 6
expensive than others. An across the board assumption of 25% damage does not take into account the cost variance of which equipment was most likely damaged. In order to arrive at what I believe to be a more accurate replacement cost estimate, I performed a more detailed analysis that considers which equipment showed the worst water spot evidence and referenced the room diagram that locates equipment placement and positioning of the worst of the down pour from the ceiling. My information on the positioning of the down pour comes from XXX personnel accounts and observations made during my survey in the security room. The Vicon (manufacturer) equipment showed the most evidence of water damage. This is in part due to the fact that it is by far the largest number of pieces as well as the fact that many of the Vicon pieces were mounted at the top of the racks. The particular Vicon products that most frequently showed heavier water spotting were the CPUs, Digital Mux, Pilot Card Cages and Video Switches. My opinion of the possibility of damage for these 4 products is no more than 50%. To arrive at what I believe to be a more accurate replacement cost I factored 50% of the replacement cost of the above 4 Vicon products and added 25% of the replacement cost for all other suspect pieces. Because Vicon provided MSRP prices on their equipment, I factored a 30% discount into the Vicon replacement costs. Working from final report of last August, there are two possible salvage values. Simple arithmetic concludes the salvage value of the undamaged equipment is the total equipment value less either the 25% Damaged replacement cost ($93,737) or the Most Likely Damaged replacement cost ($202,742). Salvage value option 1 - $374,947 (estimated total equipment replacement cost), less $93,737 (25% replacement cost) = $281,210. Salvage value option 2 - $374,947 (estimated total equipment replacement cost), less $202,742 (estimated Most Likely Damaged replacement cost) = $172,205. I believe either of those values are best case in the real world. The reasons being, 1. It has not been established definitively that any given piece of equipment is not damaged, 2. The pieces do not have cabling, an additional cost to the buyer, 3. These conditions make it inappropriate to represent the equipment as being in good quality condition, 4. Piece by piece sale is impractical in these circumstances, so the likely buyer is someone who will buy the entire lot for pennies on the dollar. I do not have a resource to assess definitively what a salvage dealer / buyer would pay, but given the vintage of the equipment, the circumstances of the equipment coming to Page 7
salvage, and the costs of making the equipment ready for sale, I would estimate no more than several thousand dollars. GOING FORWARD I understand there is a possibility the insurance carrier may want a technical comparison of the features and capabilities of the new network versus the replaced network. The objective being to make a determination that states whether XXX made a reasonable replacement choice selecting the equipment and designing the network as they did. The question may be phrased did they buy a Mercedes to replace a Corolla? If that analysis is desired, I require 40 man hours to perform that service. To be most time efficient I will need to contact the vendor of the new network to request some information. It would also be helpful if XXX could provide some engineering details about the replaced network. If the information requests cannot be facilitated, the analysis may take longer than 40 hours, or I may be limited to making statements of opinion rather than fact. Page 8