Cybersecurity: Lessons Learned from DOE Projects

Transcription

1 1 Cybersecurity: Lessons Learned from DOE Projects Mark Morgan: PNNL Ginger Armstrong: CMEEC Sue Blanchette: Groton Utilities Scott Franklin: Exelon Paul Hartung: NOVEC

2 2 Program Outline Background: SGIG Process Results/Observations Federal Role moving forward Awardee perspective

3 3 Just the facts, Ma'am Smart Grid Investment Grant (SGIG) was funded by the 2009 ARRA and managed through DOE 400+ applied 99 Grants awarded $3.4B of federal funding, matched by $4.4B of private sector funding

4 4 Just the facts, Ma'am Broad distribution of Utility Types Diverse range of Utilities (size) and award values Broad mixture of technology types Transmission (10) Distribution (13) AMI (31) Customer Systems (5) Cross-cutting (40)

5 5 The Role of Cyber in SGIG Cybersecurity requirements built in from the outset Pre-award: Addressed in proposal Post-award: Prior to work starting the awardee was required to develop and submit a Cybersecurity Plan (CSP) Annually: DOE site visit with a dedicated discussion of cyber implementation progress and alignment with approved CSP Site visits occurred from 2011 through 2014

6 6 The Cyber Team DOE requested PNNL to establish a team to support the lifecycle of the program Drew from Universities, National Labs, and private industry Established processes and conducted site visits Established resources to aid utilities

7 7 Site Visit Analysis Process Data/Grades for 13 criteria were collected and analyzed from the site visits using a green/yellow/red rating scale Demographics were collected and the awards were classified into one of 3 categories; PUD/Cites, Rural Electric/Cooperatives, and Transmission & Generation Grades and changes in grade were determined

8 8 Site Visit Results Highest performers were Transmission & Generation followed by Cities/PUD s and then Rural Electrics/Cooperatives Highest gains in performance was demonstrated by Cities/PUD s followed by Rural Electrics/Cooperatives and then Transmission & Generation

9 9 Example Data Results RE/COOP Normalized Score (%) Average 2013 Average Change in Average CP RE/COOP T&G (1.0) All Portfolio

10 10 Major Observations Cybersecurity built in not bolt on Benefits in both cost and accountability White Hat, not Black Hat as a future role model Corporate Cyber vs. Operations Cyber Benefits to industry and DOE Value of a standards based approach Senior Management accountability Vendors response The benefits of the CSP during and after the project

11 11 Sustaining the Momentum Cybersecurity is not fire and forget Large family that includes a Government role Examples of Federal roles: DOE:Electricity Subsector Cybersecurity Capability Maturity Model (ES-C2M2) NIST: Framework DHS: US-CERT NERC: ES-ISAC Excellent non-federal resources such as National Rural Electric Cooperative Association (NRECA)

12 Awardee Perspective 12

13 13 Program Project No: CYBER SECURITY PROJECT Presenters: Ginger Armstrong Sue Blanchette

14 Who We Are Groton Utilities provides electric service to two separate service territories: City of Groton, CT and Bozrah Light and Power (BL&P). GU also provides water and sewer to customers in the City of Groton. South Norwalk Electric and Water (SNEW) provides electric and water services to customers in South Norwalk, CT Jewett City Department of Public Utilities (JCDPU) provides electric and sewer services to customers in the borough of Jewett City, CT Norwich Public Utilities (NPU) provides electric, water, sewer and gas services to customers in the city of Norwich, CT Connecticut Municipal Electric Energy Cooperative (CMEEC) is a joint action agency that provides wholesale energy procurement, small capacity demand response generation, limited high voltage transmission resources and the management of energy efficiency programs on behalf of its wholesale customers 14

15 15 Demographics Characteristic BL&P GU NPU JCDPU SNEW CMEEC Customers 2,700 13,496 20,900 2,200 6, Total Employees IT Employees Peak Load (MW) Annual Energy (million KWh) ,006 Service Territory (Square Miles) Distribution Line Miles

16 16 Technical Scope ConnSMART Program Technical Scope Program Management Metering & Communications Interval Data Processing & Presentment Customer Demand Management Distribution Automation Project Management DOE Grant Administration We are Here Wholesale Business Intelligence Security Controls and Testing Cyber Security Planning & Compliance

17 17 Cyber Security in the Boardroom Executive Buy-In Company-wide Buy-In Board Education and Buy-In Plan approval Value of & Need for Awareness Training Incorporate as regularly reported CS metric at Staff and Board meetings

18 18 Vendor Readiness Smart Grid hardware still maturing Software design basis lacking in Cyber Security considerations Vendor reluctance to comply with CSPs Federal and Industry Standards slow to develop, vendors unaware of requirements Many delays due to lack of vendor readiness

19 19 Tools that made life easier Frequent Cyber Security Team meetings Team developed Cyber Security Status Tracking tool early on in project Use of outside subject matter experts Information exchange during DOE Site visits Vulnerability assessments engagements Boston Cream donuts, Strong coffee & Humor

20 20 Now what? Money & People Funding Cyber Security into the future Continued Board understanding & support Staffing Outsourcing Working without a net Managing risk, monitoring evolving threats and advancing technologies

21 21 What would you do differently? Better job of management buy-in at beginning Tighter collaboration between the vendors from the start Better understanding of resource and funding requirements for Cyber Security Vendor proof of concept Staff dedicated to project and the future

22 22 Thank You Let us rise up and be thankful, for if we didn t learn a lot today, at least we learned a little, and if we didn t learn a little, at least we didn t get sick, and if we got sick, at least we didn t die; so, let us all be thankful. Buddha

23 23 Cybersecurity Lessons Learned at NOVEC Paul Hartung, PE Manager Substation & Telecommunications

24 24 Outline Overview of NOVEC DOE Grant Background Cybersecurity Implementations Cybersecurity Roadmap Lessons Learned

25 25 Overview of NOVEC Northern Virginia Electric Cooperative (NOVEC) provides reliable electric service to more than 155,000 homes and businesses located in Clarke, Fairfax, Fauquier, Loudoun, Prince William and Stafford counties, the City of Manassas Park and the Town of Clifton, all in the state of Virginia. NOVEC's service territory encompasses 651 square miles with more than 6,880 miles of power lines. Summer peak load of ~925MW 53 Substations with Intelligent Electronic Devices (IED) communicating over fiber, microwave, radio and cellular networks.

26 26 DOE Grant Goals Substation Automation: Install modern digital control equipment to better monitor and control substation assets Feeder automation Voltage Regulator Peak Demand Voltage Reduction program Utilize DNP 3.0 Distribution Automation The VAR control project will strategically place switched capacitor banks Install remotely controlled switching devices consisting of intelligent electronic reclosers and Motor Operated switches to improve reliability utilizing two-way communications for data collection and remote control.

27 27

28 NOVEC s Cybersecurity Enhancements for SCADA 28 SGIG Project: Developed a Cybersecurity (CS) Plan Conducted SCADA Security Assessment Guiding Principle No SCADA connection to the internet Electronic and physical separation from the corporate network and from the internet - SCADA Firewall and DMZ solution Controlled internal SCADA access Implemented a manual disconnect process for remote vendor access Enhanced access controls for vendor and internal support Operational audit capability - Implemented centralized logging and monitoring access to the SCADA and Substations environments Installed and configured Intrusion Detection technology

29 29 Network Segregation Implemented DMZ architecture Eliminated direct inbound connections to SCADA Implemented more secure remote access method Includes private fiber and MPLS network

30 30 Log Management Centralized SCADA security event logs Implemented Solar Winds Log & Event Manager (LEM) Drafted log management policy and procedure Daily log review

31 31 Intrusion Detection Implemented SourceFire Intrusion Detection System at key SCADA and Substations network chokepoints

32 32 Vendor Access Controls Vendor support now requires pre-coordinated and a physical connect/disconnect process instead of full time VPN access SCADA administrators receive notifications for vendor logins

33 33 Security Patching Updated patches to remediate 2011 vulnerability assessment findings Monitor vendor patch notices and apply as required Program will be formalized under policy and procedure effort

34 34 NOVEC s Cybersecurity Roadmap DOE Cybersecurity Capabilities Maturity Model Assessment (C2M2) Completed in April 2014 and established a maturity baseline Developing a road map to accomplish maturity level 2 Cyber Training and Education Formalizing cyber education for all NOVEC employees Conduct annual Phishing Test and Cyber Survey Conduct Annual Vulnerability Assessment SCADA, Corporate, and Wireless Implementation Support for Metering, Outage Management and distribution line devices Firewall MPLS 3 rd Party connectivity Migrate OMS (Outage Management System) and MV-90 (Metering data)to DMZ

35 35 NOVEC s Cybersecurity Roadmap Create & implement processes for patching SCADA Network Substation Servers Other substation assets to include 3 rd party client applications Continually assess vulnerabilities and Identify solutions New IP based radio system Direct connection of Distributed Automation devices to fiber Implement additional access controls Security controls and firewalls

36 36 Lessons Learned Make CS integral to design Balance security vs operability DOE site visits reinforced CS Vulnerability testing proves the solution works Vendors can do more CS requirement the same but solutions may differ dependent on utility size

37 37 Cyber Security Lessons Learned Scott Franklin Manager Cyber Security Architecture and Design

38 38 Agenda Overview of BGE, PECO DOE Grant/Smart Grid Smart Meter (SGSM) Program Overview Cyber Security Program Overview Critical Success Factors Challenges The Path Forward

39 39 BGE Overview Baltimore Gas and Electric (BGE) serves more than 1.2 million business and residential electric customers and more than 655,000 gas customers located in Baltimore City and all or part of 10 Central Maryland counties BGE s electric service territory encompasses 2,300 square miles with more than 26,000 miles of both overhead and underground power lines BGE s gas service territory encompasses 800 square miles with more than 7,100 miles of natural gas pipeline mains Summer peak load of ~7,200 MW 243 substations Approximately 3,400 employees

40 40 PECO Overview Philadelphia Electric Company (PECO) serves more than 1.6 million business and residential electric customers and more than 500,000 gas customers located in southeastern Pennsylvania PECO s electric service territory encompasses 2,100 square miles with more than 29,000 miles of both overhead and underground power lines PECO s gas service territory encompasses 800 square miles with more than 6,600 miles of natural gas pipeline mains Summer peak load of ~8,250 MW 500 substations Approximately 2,400 employees

41 41 BGE DOE Grant/SGSM Program SGIG Program Goals ( Territory-wide deployment of Advanced Metering Infrastructure (AMI) assets including: 575,081 Electric Smart Meters AMI Communications Systems Meter Communications Network (RF mesh, 1,250 network devices) Backhaul Communications (Cellular) Customer Care and Billing System (partially funded by the SGIG program) Meter Data Management System Customer Web Portal and Home Energy Reports 202,906 Direct Load Control Devices 144,482 Smart Thermostats Peak-Time Rebate (Default Residential Tariff) Overall SGSM Program Goals 1.2 million Electric Smart Meters 800,000 Gas IMUs

42 42 PECO DOE Grant/SGSM Program SGIG Program Goals ( Territory-wide deployment of Advanced Metering Infrastructure (AMI) and Distribution Automation (DA) assets including: More than 600,000 Smart Meters AMI Communication Systems Web Portal Access Distribution System Automation/Upgrades Distribution Management System/SCADA Intelligent Substation Upgrades Feeder Monitors/Indicators Automated Feeder Switches Capacitor Automation Dynamic Pricing Programs Time of Use Pilot program offered to 120k Residential, 10k Commercial accounts In-Home Displays Pilot ~200 accounts Customer Education and Communication Overall SGSM Program Goals 1.2 million Electric Smart Meters 525,000 AMI Gas Module 4,400 MV-90 AMI Electric Meters

43 43 Cyber Security Program Overview Common approach for BGE and PECO DOE (and public-facing) Cyber Security Plans reflected existing, mature cyber security programs and incorporated SGSM-specific requirements as needed (NISTIR-7628, ISO, CoBIT, NIST Cybersecurity Framework, ES-C2M2) Intelligence-driven risk management approach to cyber security Intelligence and security analysis (threat models) Comprehensive risk management program Configuration and vulnerability management are critical Secure by design Build security in from the start (end-to-end project management lifecycle-integrated security, work with vendors/partners) Cyber security requirements and SLAs embedded in vendor/partner contracts, and ENFORCED Secure architecture by default Exhaustive cyber security vulnerability assessments (ad hoc, recurring), work with vendors/partners to remediate any findings Continuous, automated security monitoring and alerting (SIEM, IDS/IPS, netflow, full packet capture) across the SGSM infrastructure Security awareness training

44 44 Critical Success Factors Executive support (governance, budgetary) Security as part of the corporate culture DOE site visits reinforced the absolute need for cyber security Public Utility/Service Commission support for cyber security Vendors willing to invest in cyber security and LISTEN Detailed Cyber Security Requirements Matrices provided to vendors, embedded in contracts Enhanced product security roadmaps Vendor security assessment testing (better security QA) Continuous, automated security monitoring and alerting Incident response exercises Big data tools

45 45 Challenges Lack of formal SGSM security standards So much data big data/data analytics tools Multi-million node wireless network & traditional cyber security tools (scale, protocols) Identifying and retaining qualified security personnel Ever-changing threat landscape OT/IT collision SGSM infrastructure operating lifecycle versus cyber security lifecycle Complex systems integrations with varying security capabilities Project versus sustain mindset Vendor/product maturity

46 46 The Path Forward Continual investment in cyber security personnel and tools Annual external cyber security reviews (Public Utility/Service Commissions) Enhanced, multi-utility incident response exercises Continuous monitoring and improvement