An overview of BSNL NIB-II Project1 MPLS D R NOC. Pune

Transcription

1 An overview of BSNL NIB-II Project1 MPLS D R NOC Pune 1

2 What is a Network Operations Centre A network operations center (NOC), (NOC) is an organizational unit that: Handles day-to-day monitoring of the network Serves as a point of contact for customers, internal and external Creates, processes, and sometimes resolves events that arise in the network Therefore a NOC is the central location from which an organization monitors its computer and telecommunications networks - servers, applications, routers, switches, circuits, environmental sensors etc. 2

3 The Model A NOC has mainly three modes of functionalities ::Input Output Process 3

4 Inputs Something has happened Event sources Monitoring programs (e.g. HPOV, Netcool, netsaint, etc.) 4

5 Outputs We have taken care of what happened. Event sinks Fixing problems, handing off other issues to be resolved by others. 5

6 Process The actions in the middle while and event is being worked Ticketing systems Life-cycle of an event 6

7 BSNL s Disaster Recovery NOC at Pune BSNL has set up a D R NOC At Pune which provides an end-to-end Disaster Recovery Solution that ensures the recovery of failed business processes when a data center is lost from service. The Disaster Recovery (DR) solution supports this requirement. The objectives of the BSNL DR solution are as follows: Restoration of production services Restoration of existing application functionality 7

8 Disaster Recovery : Disasters are inevitable but mostly unpredictable, and they vary in type and magnitude For an enterprise, a disaster means abrupt disruption of all or part of its business operations, which may directly result in revenue loss. The causes can be natural or human or mechanical in origin, ranging from events such as a tiny hardware or software component's malfunctioning to universally recognized events such as earthquakes, fire, and flood. Effects of disasters range from small interruptions to total business shutdown for days or months. When a disaster strikes, the normal operations of the enterprise are suspended and replaced with operations spelled out in the disaster recovery plan. 8

9 9

10 NOC status at normal condition 1

11 NMS switchover to D R NOC Pune 11

12 12

13 PMS switchover to D R NOC Pune 13

14 NMS switchover to Primary NOC Bangalore 14

15 PMS switchover to Primary NOC Bangalore 15

16 Procedure for Controlled switch over ::- Check the health condition of all the systems. Disable the respective resource from the desired cluster. Stop the applications. Perform the DB conditioning in all the sub-systems in both Bangalore and Pune. Start the applications in Pune D R NOC servers. The reverse action will be performed to bring back the applications in Bangalore NOC 16

17 Procedure for Unplanned fail over ::- Log into Pune DB server as root for NMS and to OVSA server for PMS. Start the applications one by one by calling respective scripts. Check Pune subsystem. 17

18 Procedure for Recovery of Primary NOC:NOC:- Enable Bangalore NOC (D R) by following certain procedure. Install Operating System (Oracle). Install all the NMS applications. Check the connectivity between the NOCs Check Pune subsystem. Recreate the Standby DB in Pune sub-system after the Primary NOC is recovered. The standard Planned switch over procedure may be followed for making Bangalore NOC as Primary. 18

19 Control measures in recovery plan Preventive measures - These controls are aimed at preventing an event from occurring. Detective measures - These controls are aimed at detecting or discovering unwanted events. Corrective measures - These controls are aimed at correcting or restoring the system after disaster or event. 19

20 Disaster Management measures ::9 sets of 6 KVA inverters providing supply to various servers and routers. Each set of inverters consists of two inverters working on load sharing basis for enhanced redundancy. There are three ac inputs given to each and every servers from three different inverter supply to ensure uninterrupted service. Temperature rise alarm system (audio and visual) has been installed inside the server room for taking immediate necessary action in case of rise of temperature. Preventive maintenance of servers, routers and switches are being performed on each quarters in a year. Constant health checking of the servers and network elements centrally from Primary NOC for any inconsistent behavior. 2

21 BSNL s MPLS NETWORK OPERATION CENTRES 21

22 MPLS NOC AT BANGALORE 22

23 MPLS DR NOC AT PUNE 23

24 Staff details : Mr P Bhattacharya, DE, MPLS/MLLN D R NOC Pune Mr. Mahesh Kothare, SDE ( Look after) Mr. Kiran Kumar Totakuri, JTO Mr. Ram Krishna Tiwari, JTO Mr. Pavan Chaudhary, JTO Ms. Khushboo Rani, JTO Mrs. Rupali Sinnarkar, JTO Mrs. Malve, JTO Mrs. Anagha Rajwade, TTA Mr. Rajendra Pillay, T/M 24

25 List of servers present in NOCs S.N 1 2 SERVERS DATABASE (SUN FIRE V89) DATABASE (SUN FIRE V89) APPLICATION RUNS, DATABASE CONTENTS. Maintaining Database (Cluster Server Oracle-9i, Secondary DB Server Node-1) for NIB2P1 Node1 Maintaining Database (Cluster Server Oracle-9i, Secondary DB Server Node-2) for NIB2P1 Node2 Regional Data Collection (Cluster OVPI-IUM, RPOP( Database for Server Node-1) RPOP Servers.) Regional Data Collection (Cluster OVPI-IUM, RPOP( Database for Server Node-2) RPOP Servers.) FUNCTION 3 RPoP (SUN FIRE V88) 4 RPoP (SUN FIRE V88) 5 RWAN (SUN FIRE V21) For storing the System logs. CISCO WORKS. 6 HPOV SIP (SUN FIRE V88) Portal Server for Customer Link Utilization and Availability. HPOVSIP HPOV NNM (SUN FIRE V88) HPOV TeMIP (SUN FIRE V88 ) HPOV PMS (HP9 RP444) MP HPOV TeMIP AM (SUN FIRE V88) Link Monitoring through Alarms browser. Link Monitoring through Alarms browser Customer Link Creation Maintaing Multiplay Users Access Statistics Forwarding Prsentable Multiplay MP HPOV TeMIP PM Users Access Statistics to Portal (SUN FIRE V88) Server MP HP OVPI (SUN FIRE Multiplay Users Performence V88) Monitoring Database MP HPOV SIP (SUN Multiplay Portal Servers User Access FIRE V88) Reports NNM HPOVTeMIP HPOVPMS HPOVTeMIP AM HPOVTeMIP PM MP-HP-OVPI MP-HP-OVSIP 25

26 Different applications used at NOC Network Node Manager ( HP OV NNM) Service Activator ( HP OV SA) Performance Insight ( HP OV PI) Subscriber Information Portal ( HP OV SIP) TeMIP ( Telecom Management Information Portal) MRTG ( Multi Router Traffic Grapher) Grapher) Free ping Remedy ( Runs in ProjectProject-3 Platform) 26

27 List of ProjectProject-1 Routers and switches in Pune:Pune:Name of the Equipment Core router Description Metro Core router Cisco-1241 LAN Switch Cisco-659 Cisco No. of equipments & location One in MKR Two (One each in MKR & MHS) Two in MKR Provider Edge router Cisco-7613, Three ( Two in MHS & one Juniper M-4, in MKR Juniper M32 CDR PE Routers Cisco-7613 Two ( both are in Chinchwar IT Project) 27

28 28

29 Main Responsibilities of NOC Provisioning :- Rolling out services to the new customers and modification of the existing links as and when required. Intra-AS Network Management :- Backbone link + Customer link. Inter-AS network management :- IGW, IXP links. Performance Management :- Backbone + Customer links. L-1 support :- Booking & escalation of faults L-2 support:- Technical support to field units. L3 Support :- Configuration analysis and fine tune performance of links. Technical support to EB teams as and when needed. 29

30 Activities at D R NOC Pune Initially D R NOC Pune started with doing Proactive monitoring of 19 VVIP Customers having more than 26 circuits. Very recently with the instruction and guidance of the GMM Mumbai, the following activities are being carried out from D R NOC Pune :Fault handling and escalation for Western Region Nodes. ( Presently from 8 am to 8 pm) IGW link monitoring for Mumbai during day time. Backbone link monitoring from 8 am to 8 pm. L2/L3 support for western region customers and handling internet issues. Call Handling and providing technical support to BD cell of western region. Weekly and monthly backbone link performance report generation along with Node cut-off issues and generation of any other report required by the higher authorities as and when asked for. Monitoring of IPSTAR LINKS OF. 3

31 Statistical Data 31

32 BSNL MPLS Nodes Category of Cities No. of Core routers No of Edge routers No. of Metro Core router A1 1 (-2) 18 (-4) 6 (- ) A2 A3 A4 B1 B2 3 (-2) 1 (-7) 6 (-4) 6 (-1) 9 (-1) 1-2) 1 (-2) 2 (-1) 2 (-5) 3 (-1) 62 (-13) 129 (-32) 12(-4) TOTAL 34 (-9) No. of CDR No of IGW No of IGW No of PE router PE routers Misc routers 2 (-) 5(-1) 5 (-1) 11 (3) 4 (-2) 2 (-) 8 (-2) 5-1) 5(-1) 11 (-3) 32

33 33

34 Core Routers of with location type S.no Location Location Type Region Circle Host Name 1 Mumbai A1 mum-rtr-pp-1 2 mumbai A1 Maharashtra mum-rtr-pp-2 3 Nagpur A4 ngp-rtr-pp-1 4 Pune A2 pun-rtr-pp-1 5 Ahmedabad A2 Gujarat ahd-rtr-pp-1 6 Rajkot B1 Gujarat rjt-rtr-pp-1 7 Indore A3 Madhya Pradesh ind-rtr-pp-1 8 Jabalpur B2 Madhya Pradesh jbl-rtr-pp-1 9 Raipur A4 Chattisgarh ryp-rtr-pp-1 34

35 LIST OF PE ROUTERS OF SNO Location Ahmedabad ahmedabad Ahmedabad Anand Surat Vadodara Mehsana Rajkot Sagar Satna Bhopal Gwalior Indore Jabalpur Dhule Kalyan Kolhapur Aurangabad Mumbai Mumbai mumbai Nagpur Nanded Nashik Panjim Pune Pune Pune Pune Solapur Raipur Bilaspur Location Type A2 A2 A2 B2 B1 B1 B2 B1 B2 B2 B1 B1 A3 B2 B2 B2 B2 B2 A1 A1 A1 A4 B2 B1 B2 A2 A2 A2 A2 B2 A4 B2 Region Circle Host name Gujarat Gujarat Gujarat Gujarat Gujarat Gujarat Gujarat Gujarat Madhya Pradesh Madhya Pradesh Madhya Pradesh Madhya Pradesh Madhya Pradesh Madhya Pradesh Chattisgarh Chattisgarh ahd-rtr-pe-1 ahd-rtr-pe-2 ahd-rtr-pe-3 and-rtr-pe-1 srt-rtr-pe-1 vda-rtr-pe-1 msn-rtr-pe-1 rjt-rtr-pe-1 sag-rtr-pe-1 sat-rtr-pe-1 bpl-rtr-pe-1 gwl-rtr-pe-1 ind-rtr-pe-1 jbl-rtr-pe-1 dhl-rtr-pe-1 kln-rtr-pe-1 klp-rtr-pe-1 ang-rtr-pe-1 mum-rtr-pe-1 mum-rtr-pe-3 mum-rtr-pe-4 ngp-rtr-pe-1 nnd-rtr-pe-1 nsk-rtr-pe-1 pjm-rtr-pe-1 pun-rtr-pe-1 pun-rtr-pe-2 pun-rtr-pe-3 pun-rtr-pe-4 slp-rtr-pe-1 ryp-rtr-pe-1 bil-rtr-pe-1 TOTAL NODES

36 SNO LIST OF METRO CORE ROUTER IN Host name Location Location Region Type Circle 1 ahd-rtr-mcr-1 Ahmedabad A2 Gujarat 2 ahd-rtr-mcr-2 Ahmedabad A2 Gujarat 3 pun-rtr-mcr-1 Pune A2 4 pun-rtr-mcr-2 Pune A2 36

37 LIST OF CDR PE IN SNO Host name Location Location Type Region Circle 1 pun-rtr-cdrpe-1 Pune A2 2 pun-rtr-cdrpe-2 Pune A2 37

38 LIST OF IGW/IGWPE IN 1 mum-rtr-igw-1 Mumbai A1 2 mum-rtr-igwpe-1 Mumbai A1 38

39 LIST OF MISC ROUTERS IN SNO Host name Location Location Type Region Circle 1 mum-rtr-iptax-1 Mumbai A1 2 mum-rtr-irr-1 Mumbai A1 3 mum-rtr-ixp-1 Mumbai A1 39

40 SUMMERY OF NIB-II BACKBONE LINKS TYPE OF LINKS TOTAL (ALL OVER INDIA) TO TO OTHER REGION STM STM GIGE GRAND TOTAL

41 41

42