Support Guide Comprehensive Hosting at Nuvolat Datacenter 1
Table of Contents Introduction... 4 Roles, Responsibilities & Expectations... 4 Customer... 4 Nuvolat Support... 5 Deliverables... 5 Best Practices... 7 Product Manuals... 8 Product Trouble Shooter... 8 Product Knowledge Base... 8 Support Request Types... 8 Emergency Support Request... 8 Regular Support Request... 8 Information Request... 8 Alert Classification... 9 Expected Response and Resolution Time... 9 Support Line Knowledge Matrix Nuvolat and Partners...10 How to contact the Nuvolat Support team....12 Tickets... 12 Chat... 12 Email... 12 Phone (Regular Support)... 12 Phone (Emergency Support)... 12 Types Defined (Emergency Issues):... 12 Emergency... 12 Regular... 12 Information... 13 SLG (Service Level Guarantee) Exclusions...13 Problem Management Priority Guideline...14 Disaster Recovery (Only with vrun DR Pack / vrun Comprehensive Hosting Services)...15 DR Testing...15 DR Activation Process...15 IMPORTANT REMARK... 16 2
Expectations...17 Before creating Support Request:... 17 You are expected to:... 17 Emergency Maintenance Window...17 Summary and Conclusion...18 3
Introduction Nuvolat has created an extensive product documentation that has all functionalities explained in detail. In the unlikely event of something being unclear or not functioning as expected you can contact the Nuvolat support team. Before contacting the Nuvolat support team we do advise you to consult your product manuals and troubleshooter. Roles, Responsibilities & Expectations Customer System Administrators at the partner Side / client side are responsible for collecting and submitting the necessary information for each support session / support ticket when its an inbound support request. System Administrators at the partner side / client side should set the correct expectation regarding resolution times with their customers / end users. System administrators / clients are requested to go through required best practices to make sure they are not over provisioning the system OR making any changes at storage / hypervisor level that are unknown to Nuvolat support team. Its important for System administrators / clients to consult Nuvolat support team before making any changes to default configuration of vrun / Hypervisor / Firewall. Incase vrun hardware appliance is procured by partner (wherein Nuvolat is only providing vrun Software and Comprehensive Hosting for vrun servers), we shall not maintain any spares at Datacenter. Partner needs to provide these spares for emergency hardware replacements. In such case, our team shall help you identify if it s a hardware issue and shall also perform replacement of hardware by coordinating with datacenter team, but RMA / procurement of new part will be managed by partner. If this involves shipment of faulty spare outside datacenter, shipment will be charged as per actual to partner. 4
Nuvolat Support Our support teams will manage the vrun environments 24x7 that are under contract agreement of Comprehensive Hosting Services and hosted at one of our Datacenters. Nuvolat Support team will be accountable for end to end support for vrun stack which includes managing alerts, proactive checks, diagnosis and troubleshooting of problems. Our support team shall perform periodic maintenance activity (no physical movement) of vrun stack with prior consent from partner. Comprehensive Hosting Services also includes remote hand support at our datacenter, which means should we require any assistance in replacing hardware / performing remote activities, it will be handled by our support team with datacenter support. If the vrun appliance (prefabricated) unit is procured / purchased from Nuvolat, our support team shall also manage the RMA replacement of hardware. In such case we shall also maintain spare hardware components like disk, power supply for emergency replacements. Support team shall also manage, periodic updates of Hypervisor and upgrades of vrun software with prior consent from partner and with planned downtime. Deliverables Below are the list of deliverables which is part of Comprehensive Hosting Services agreement. Nuvolat Support Partner / Customer General 24x7 Monitoring and Management Nuvolat Ticketing System Customized Ticketing System Helpdesk with phone support Sysprep OS Templates Custom OS Templates Custom application installers 5
Network and Security Storage & Compute VLANs Firewall and Load Balancer (Virtual Appliance) Public IPs Switch Ports Gateway / Router (Virtual Appliance) DNS Management IP whitelisting Burst Storage Capacity for replication (as per actual price offered to you) NAS Share to upload images Filer / SAN for additional backup Burst Storage Capacity for expansion (as per actual price offered to you) Burst Compute Capacity for expansion (as per actual price offered to you) External USB to copy images Installations and Upgrades Replacements Installation and Upgrade of an Operating System Installation and Upgrade of an Application Hardware installation of vrun appliance and relevant components Hardware installation of add on devices Hypervisor installation and upgrades vrun installation and upgrades Firewall and Network installation and configuration 6
Replacement of faulty component where vrun appliance (prefabricated is provided by Nuvolat) Replacement of faulty component where vrun is only provided as software along with Comprehensive Hosting Services and hardware is procured by partner. Licensing Procurements OS License VMWare Licenses Application License vrun License GMC License Procurement of spares where hardware is purchased from reseller other than Nuvolat Procurement of software patch, applicable software upgrades for other than vrun, NAS Share and Firewall Appliance Procurement of spares where hardware is purchased from Nuvolat Best Practices Our records show that partners / clients who have invested the necessary time in documentation and training modules often experience minimum resolution time on support sessions. This is usually because well-trained and informed partners can collect all relevant information and can help troubleshooting the environment that presents the issue. Before installing / using our products it is always advisable to read the corresponding release notes and other technical product documentation. We also advise to read through this support guide and to make sure your system administrators are aware of the Nuvolat support procedures. This documentation explains you step-by-step how to setup, manage and monitor your vrun product through the CMC and all additional tools. Version No: 1.4.0 7
Release Type: General Availability (GA) Essentials : Before starting to sell and implement vrun Quick Reference Guide : Quick guide on how to use the vrun Installation Guide : How to get the vrun hardware up and running User Guide : The CloudFrames vrun user and administrator documentation Best Practices : CloudFrames vrun Best Practices for correct functioning and performance Release Notes : Release notes and known issues Product Manuals vrun.nuvolat.com/documentation Product Trouble Shooter http://tickets.nuvolat.com/index.php?/troubleshooter/list Product Knowledge Base http://tickets.nuvolat.com/index.php?/knowledgebase/list Support Request Types Emergency Support Request Emergency Support can be requested for events resulting into production Down Time of vrun (including Virtual Machines) or events where Data Loss is experienced. Regular Support Request Regular Support can be requested for all issues where no Down Time and No Data Loss is experienced but Product does not behave as expected. Information Request Information Requests can be filed after carefully consulting the Product Manuals, Trouble Shooter and Knowledgebase. 8
Alert Classification Critical Alert Events resulting into production downtime of vrun including Virtual Machines. Events where potential Data loss is experienced are also considered as critical. Non-Critical Alert Events resulting into intermittent issue, anything that is not affecting production environment / production loads in anyway are considered as Non-Critical. Warnings Any non-urgent support request for information, one time notification from vrun system etc. results in a warning / notification. Expected Response and Resolution Time Expected Response Time / Turnaround time after submitting request (*): Acknowledgement of the receipt of the support call is offered within the timings listed below. Expected Response / Revert Time Critical Alert / Emergency Situations Non Critical Alert Warnings 15 Min 1 hour Best Effort Support Channel Call to TL / TM (Emergency ONLY) Regular Chat / Ticket / Email Regular Chat / Ticket / Email PS: Submitting ticket / chat procedures are further described further in this guide. Expected Resolution Time / Turnaround time with proposed solution / PoA (*): In order to start resolving the issue, the 1 st Line support might require additional information from you. If this information is not available our NOC support engineers cannot start pursuing a solution and will inform you about the missing topics. Based upon the information that is offered by you and/or collected by our Support Teams, a solution will be offered. We always aim to reach the resolution times as listed below. Expected Resolution Time Critical Alert Non Critical Alert Warnings 24 hours 48 hours 72 hours 9
(*) Nuvolat is highly committed to its customers and will spend the necessary efforts to achieve all response and resolution times. Unfortunately we cannot always guarantee the resolution timings listed above. Support Line Knowledge Matrix Nuvolat and Partners General Networking Storage Operating Systems ERP software Relational Databases Mail Servers/Clients Terminal/Server setups Third Party Application - VDI 1st Line Nuvolat Support 2nd Line 3rd Line ENG Basic TCP/IP + DHCP Firewall & Load Balancers Static routing Dynamic routing VLAN SATA/SAS/SCSI iscsi Partitioning / File Systems RAID Concepts (concepts of) distributed storage Basic BitSpread knowledge Advanced BitSpread knowledge Basic Volume driver knowledge Advanced Volume driver knowledge Basic Microsoft Windows desktop Partner / Customer System Admin 10
Hypervisor Backup and Restores Virtualization Pylabs Framework Advanced Microsoft Windows desktop Basic Microsoft Windows server Advanced Microsoft Windows server Information of Required Device Drivers Windows boot order and support tools Linux basic knowledge Linux advanced knowledge Linux networking Linux troubleshooting Linux performance monitoring Hypervisor concepts Advanced Hypervisor Troubleshooting Basic Backup concepts and archiving Advanced backup concepts Basic restore and troubleshooting Advanced restore and troubleshooting Virtualization Concept Physical to Virtual Migrations Virtual to Physical Migrations Basic python scripting Advanced python scripting Basic Framework knowledge Advanced Framework knowledge Basic Arakoon knowledge Advanced Arakoon knowledge 11
How to contact the Nuvolat Support team Tickets http://tickets.nuvolat.com/ The Nuvolat Ticketing portal requests login. In case you would not have credentials, please contact your sales rep or send email to tam@nuvolat.com / sm@nuvolat.com Chat Our chat support is based on a unique invitation code. vrun support engineer shall provide you rescue session code in if we need to share remote session for troubleshooting purpose. Email tam@nuvolat.com For presales and accounting vrunsupport@nuvolat.com Production Support vrun@nuvolat.com - Sales Phone (Regular Support) (408) 622 08 56 (408) 622 04 53 Phone (Emergency Support) (408) 940 69 41 Types Defined (Emergency Issues): Production VM Down: Any virtual machine hosted on vrun, may it be native VM, failover virtualization by promoting replica VMs, VM on hosted vrun is down which is further defined as completely unusable OR performing as slow as unusable. vrun Host Down: Any issue that causes complete unresponsiveness to the vrun host OR causing sluggish performance that is as slow as unusable. Integration issues with Node / any core application / component failure that may put vrun availability in jeopardy. 12
Failover Virtualization / Migration: Any failure while promoting replica VM to production OR performing physical to virtual migration. Backup Backlog: Constant backup / snapshot backlogs / snapshot failures (more than 24 hours backlog / more than 24 hours constant backup failures, may it be local snapshot / replication can be considered as emergency situation. Emergency Regular Information Ticket * * * Chat Phone A ticket needs to be created for every support request. That to allow a clear communication and information sharing. SLG (Service Level Guarantee) Exclusions The following periods of time represent exclusions from the SLG s: 1. Periods of scheduled and emergency maintenance as defined in this guide and any other times that may be specifically agreed to with Customer. 2. Periods of downtime due to Customer-provided content or programming errors. Example: a CGI script or program written by Customer, or contracted party, that causes the server to be rebooted or Hypervisor to fail. 3. Periods of downtime due to Customer-initiated commands, file transfers, or systems administration. 4. Periods of downtime due to Customer directed and requested work that is not related to the routine management of Services. 5. Individual server or network component outages that do not impact the overall availability of the Service due to redundancy in the design. 6. Evaluation or approval of new software or hardware for use within Nuvolat data center. This includes systems developed outside of Nuvolat such as third-party systems or systems developed by Customer. 7. An outage that is due to a DOS (Denial of Service) attack 8. An outage that is due to a worm or virus infection 9. An outage that is due to a vulnerability in the Customer software 10. An outage that is due to Customer application errors 11. An outage that is due to instructions by Customer that is not related to the routine management of Services. 13
12. The number of users exceeds the capacity of the infrastructure and application 13. Acts of God OR natural calamities. Business Hours: Monday to Friday 8:00 AM EST to 5:00 PM PST. Support Hours: 24x7 For Emergency Support Requests we have a number of engineers on call 24/7. Regular and Information Requests can only be handled during Business Hours. Problem Management Priority Guideline Priority Level* Description** Examples Work Begins 1 CRITICAL Immediate attention required. Complete failure of critical services or hypervisor causing the web service to be completely unavailable. Any failure that causes a severe or significant impact including failures, lack of performance and/or frequent reboots to an extent that applications running on virtual machine(s) are unavailable for practical business purposes. Network or power outages or disasters. Hypervisor non-functional and not able to run one or more virtual machines. Service failures including failures that prevent servers to run on the hypervisor Emergency Maintenance. Within 15 minutes 2 MAJOR Attention required as soon as possible. Partial failure of critical services or applications. A failure that may not have an immediate effect but could lead to the unavailability of web service or server uptime. Prolonged network degradation including latency, packet loss, or off-network failures. Access problems with application, data, or other systems due to password or profile problems. Servers are running but no ability to deploy additional servers on top of the hypervisor Within 30 minutes 14
3 MINOR Complete or partial failure of non-critical services or applications with minimal Customer impact. Identified issues that do not impact Customer services. Routine Network Management (e.g. NAT translations, firewall rules and open ports) Remote Access troubleshooting. Remote Hands set-up and troubleshooting. Billing questions. Other unsupported Customer requests. Within 24 hours Disaster Recovery (Only with vrun DR Pack / vrun Comprehensive Hosting Services) DR Testing By default provisional compute capacity (8VPUs 16GB RAM 500GB Storage 1200 IOPS) is available and can be assigned instantly against each vrun environment for DR testing. There is no hard coded limit on number of times partner can test DR by virtualizing / performing test promote of replica VMs, although we recommend to perform it once a month. Network: Please note that, during test promote test VMs will be provisioned in separate VLAN and will not have public access, it can be accessed through GMC console to check for data integrity. Please refer to GMC product documentation on how to access replica VMs. DR Activation Process Important: Please refer Reseller Pricing Sheet on Nuvolat Partner Page for service charges of DR activation and exact deliverables. Intimating / Declaring DR: DR situation can be called by partner under different circumstances. DR SLA / SLG starts from the time of intimating DR to Nuvolat support team. Our turnaround time / response time on DR request will be 15 minutes provided the right method of communication is followed. We shall put in our best effort and initiate required logistics to bring up your production environment to live as fast as we can. 15
Email ID: dractivation@nuvolat.com Phone: (408) 940 3484 Please note that DR activation phone line is only meant for specific purpose, any other support request related to onsite emergency / regular support will not be entertained on this number. Please check earlier section in support guide on how to avail emergency support. Once we receive your DR request, our support team shall call you back / write you back to get formal approval on DR activation charges. Post approval, DR activation process will begin which involves: 1) Provisioning required compute capacity in shared cluster to virtualize machines. 2) Providing required number of VLANs in shared cluster network 3) Promoting replica VMs to life (replication will break as replica instance will come to life). This can be also perform by partner. 4) Restoring firewall configuration from old firewall instance to new firewall instance 5) Assigning required number of Public IPs on firewall 6) Making sure, all VMs are up and running within defined RTO (2 Hours minimum) and RPO (30 minutes minimum) 7) Post hardware replacement / disaster scenario, configuring DR VMs to replicate back to original source. 8) Post replication, bringing VMs back to live on source vrun environment. 9) Redoing / Changing network configuration and restoring firewall configuration. DNS Redirection: Post DR activation DNS redirection need to be perform by partner. Since we are not hosting DNS server managing DNS records for any partner, this need to be done by contacting your local DNS provider / vendor who manages DNS for you. Our support team shall help you to provision required public IPs for your workloads, you need to provide these IPs to your DNS provider to configure right DNS mapping. IMPORTANT REMARK Out-of-Business-Hours support is reserved for Emergency Support Requests. We expect our partners to use on-call support resources and out of business hours support only in explicit conditions / issues that cannot be hold back till next business day. 16
Expectations Before creating Support Request: Please check to see if: Any of the Nuvolat product documentation explains and solves the issue. The network switches are operational and are Gigabit switches The vrun environment is running in an adequately cooled environment. Preferably should be rack mounted in a server room or datacenter. The vrun environment has stable internet connectivity The correct firewall rules are applied for vrun to connect to internet Product Manuals vrun.nuvolat.com/documentation Product Trouble Shooter http://tickets.nuvolat.com/index.php?/troubleshooter/list Product Knowledge Base http://tickets.nuvolat.com/index.php?/knowledgebase/list You are expected to: Collect and submitting the necessary information for each support ticket. Set the correct expectation regarding resolution times with your customers / end users. Spend a best-effort attempt to resolve the issue using all published Nuvolat supporttools and documentation sets. Emergency Maintenance Window Nuvolat is committed to keep your device performing at a maximum level and shall provide hot fixes even when your environment / device did not suffer from a specific problem. These fixes are mandatory and might require downtime for your environment / device. Should you choose not to allow deployment of these hot fixes, Nuvolat cannot support your environment. 17
Summary and Conclusion The support procedures and specific support instructions within this guide are intended to make sure that your overall experience with Nuvolat support team goes seamlessly without any problem. At any given point we do not wish to program our customers / block them within procedures to make user experience cumbersome. Based on ORG structure and support lines defined within NOC; these support outlines are formed by us and we assured best attention and in time resolution from Nuvolat teams when these guidelines are followed. 18