OSMOSIS Open Source Monitoring Security Issues HACKITO ERGO SUM 2014 / April 2014 / Paris
AGENDA Who are we? Open Source Monitoring Software Results Demonstration Responses Mitigations and conclusion 4/25/14 2
DEUTSCHE TELEKOM PROFILE COSTUMERS & MARKETS FACTS & FIGURES Customers >141 m mobile customers >32 m fixed-line customers/ >17 m broadband customers rd. 3 m (IP) TV customers About 2 m workstation systems marketed Markets Presence in 50 countries Deutschland, Europa, USA: using our own infrastructur T-Systems: globale presence & alliances via partners Source: DT annual report to shareholders 2012/TMUS annual report to shareholders 2012 Telekom in figures Revenue 58.7 bn Adjusted Ebitda 18.7 bn Free cash flow 6.4 bn Among the top100 companies worldwide (#75 in 2012 Fortune500 list) Employees & responsibility Employees worldwide: 235, 000 9,000 trainees und cooperative degree students in Germany Pioneer of social issues (pomotion of woman, data privacy, climate protection etc.) 4/25/14 3
DEUTSCHE TELEKOM GROUP INFORMATION SECURITY Security levels Security strategies Standards Consulting Innovation Security requirements Security requirements Privacy & Security Assessment (PSA) Deutsche Telekom Cyber Emergency Response Team (CERT) Implementation of measures Technology Testing Abuse-Handling Incident management Intelligente Netzlösungen 4/25/14 4
OPEN SOURCE MONITORING SOFTWARE OVERVIEW SUMMARY Critical function in a corporate network Lets you know how well the network is running End-to-end monitoring for services up to detailed hardware view JOINT FUNCTIONS IN THIS CASE Web based solution Agent based OUT OF SCOPE No IDS / IPS No commercial solutions No security monitoring 4/25/14 5
OPEN SOURCE MONITORING SOFTWARE THREATS Ubiquitous component in network environments Centralized access to multiple networks Usually position deep in the internal network (as in: semi-trusted network) Used in nearly each environment (from small business, over mid range up to enterprises) MTAACA (machine that acts as client attack) and CTAMTAACA (clients that access machines that act as clients attack) 4/25/14 6
OPEN SOURCE MONITORING SOFTWARE RISKS A more valuable target than perimetric systems Input data parsing (logfiles, SNMP, traps,...) Web GUIs (OWASP Top 10 anyone?) Some have home-brew agents on EVERY system Potential access to a lot of components in the perimeter and internal network 4/25/14 7
OPEN SOURCE MONITORING SOFTWARE HOW IS IT IMPLEMENTED TYPICALLY? OWN CHECKS SNMP 4/25/14 8
OPEN SOURCE MONITORING SOFTWARE WHAT WE COVERED This is not an academic talk - we are talking about actual experience Open Source tools are easy to audit (kinda) Everyone has the chance to audit their own solution Focus on market leading / industry standard software 4/25/14 9
OPEN SOURCE MONITORING SOFTWARE WHAT WE DID NOT COVER No commercial / closed source solutions Architectural software flaws Critical features which should be disabled anyways e.g. nrpe.cfg dont_blame_nrpe No additional plugins, features, add-ons Not the (home brewed) agents itself 4/25/14 10
OPEN SOURCE MONITORING SOFTWARE TOOLS WE COVERED CACTI network graphing solution ; frontend is completely PHP driven src: http://www.cacti.net NAGIOS Nagios Is The Industry Standard In IT Infrastructure Monitoring src: http://www.nagios.org/ CHECK_MK (NAGIOS ADD-ON) Check_MK is a comprehensive add-on for the famous Open Source monitoring software Nagios src: https://mathias-kettner.com/check_mk_introduction.html ICINGA Icinga is an enterprise grade open source monitoring system src: https://www.icinga.org/ 4/25/14 11
OPEN SOURCE MONITORING SOFTWARE PUBLICLY KNOWN INCIDENTS CVE2012-096 Remote Buffer Overflow Nagios Hetzner (06/2013) 4/25/14 12
OPEN SOURCE MONITORING SOFTWARE OTHER INTERESTING INFORMATION Public Buffer Overflow in CACTI (since 10/2013) NRPE - Remote command exec (04/2014) 4/25/14 13
RESULTS OVERALL Critical issues were found in ALL audited solutions Memory corruption Buffer/Heap Overflows Off-by-one s CSRF XSS eval-processing untrusted input Remote Code Execution Arbitrary file access Many web based bugs, as all the solutions use web GUIs 4/25/14 14
RESULTS DETAILED VIEW (Cacti) Version 3.5.0b 1.9.1b 1.2.2p2 0.8.8a Number of Findings 1 2 7 3 CVSS 2 Score (highest finding) 4.9 AV:N/AC:M/Au::S/C:P/I:N/A:P 8.5 AV:N/AC:M/Au:S/C:C/I:C/A:C 8.5 AV:N/AC:M/Au:S/C:C/I:C/A:C 8.5 AV:N/AC:M/Au:S/C:C/I:C/A:C Criticality medium high high high Number of open findings 1* 0 1** 3 Announcement to vendor / developer 5th Dec. 2013 2nd Dec. 2013 8th Oct. 2013 15th Oct. 2013 Bug Fix Release 3.5.x*, 4.0.3 Public DTAG CERT Advisory Remarks DTC- A-20140324-004 * Bug fixes in the source code only available. No updates release available. 1.10.2, 1.9.4, 1.8.5 or latest release DTC- A-20140324-003 1.2.4p1, 1.2.5i2 or latest release DTC- A-20140324-002 ** exec of python code within WATO n/a DTC- A-20140324-001 Confidential Christian Sielaff / OSMOSIS 03.04.2014 15
DEMONSTRATION CAN WE GET A SHELL? 4/25/14 16
DEMONSTRATION NETWORK OVERVIEW Hacker Terminal Server Cacti / Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 17
DEMONSTRATION CACTI Bugs: cross site request forgery command like exec Hacker Cacti Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 18
DEMONSTRATION CACTI Bugs: cross site request forgery command like exec Get executed on Cacti server if: Administrator clicks on a link or Visit a malicious web site Hacker Cacti Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 19
DEMONSTRATION CACTI Bugs: cross site request forgery command like exec Get executed on Cacti server if: Administrator clicks on a link, or Visit a malicious web site Pro: Get a shell Con: Need to know the Cacti URL Admins needs to access link or site with link to trigger exploit Outgoing connections my be restricted Admins needs to logged in Hacker Cacti Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 20
DEMONSTRATION CACTI Bugs: cross site request forgery command like exec Get executed on Cacti server if: Administrator clicks on a link, or Visit a malicious web site Pro: Get a shell Con: Need to know the Cacti URL Admins needs to access link or site with link to trigger exploit Outgoing connections my be restricted Admins needs to logged in not really let s brute force the Admin account J Hacker Cacti Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 21
DEMONSTRATION CHECK_MK Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 22
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 23
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What is the problem: Exploits a feature in WATO Uploads and exec a snapshot Snapshot contains plain python code Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 24
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What is the problem: Exploits a feature in WATO Uploads and exec a snapshot Snapshot contains plain python code Pro: Get a shell Con: Need to know the Check_MK URL Admins needs to access link or site with link to trigger exploit Outgoing connections my be restricted Admins needs to logged in Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 25
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What can we do better? Use the agent on a system Re-use existing connections Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 26
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What can we do better? Use the agent on a system Re-use existing connections Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 27
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What can we do better? Use the agent on a system Re-use existing connections Pro: Get a shell URL is no longer needed Administrator not need a link to click Triggers when the Administrator logs in Using existing connections Con: Need (privileged) access to a monitored system Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 28
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What can we do also? Just a simple SSH login? Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 29
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What can we do also? Just a simple SSH login? A XSS triggers a CSRF triggers an Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 30
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What can we do also? Just a simple SSH login? A XSS triggers a CSRF triggers an upload that triggers a shell J Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 31
DEMONSTRATION CHECK_MK Bugs: cross site request forgery command like exec cross site scripting What can we do also? Just a simple SSH login? A XSS triggers a CSRF triggers an upload that triggers a shell J Pro: Get a shell URL is no longer needed Administrator not need a link to click Triggers when the Administrator logs in Con: Logwatch feature (default installation is fair) Outgoing connections my be restricted Hacker Terminal Server Check_MK Administrator Confidential Christian Sielaff / OSMOSIS 03.04.2014 32
DEMONSTRATION CAN WE GET A SHELL? YES J 4/25/14 33
RESPONSES CONTACT AND TIMELINES CONTACTING some developer without a contact option (expect a public mailing list is this a good idea in such case?) usually an Email contact is possible also with a privacy option Only Icinga provides an option for a private information sharing http://www.icinga.org/faq/how-to-report-a-bug/#securityissue TIMELINE approximately six days from first response to a bug fix release well done! up to 85 days to a bug fix release up to nothing until now L ADVISORIES post flaws to Bugtraq on 24 th of March get first responses regarding open findings 28 th / 31 st of March 4/25/14 34
RESPONSES FEEDBACK WHAT IS OWASP?" It s 2014, guys! THIS IS A FEATURE" Yes, and a backdoor! WHAT TOOLS DID YOU USE FOR SCANNING? Hint: None, we had the source code Duh! WHY SHOULD WE FIX WHAT YOU SEE AS A SECURITY PROBLEM? WE NEVER ASKED FOR THIS AUDIT! Approximately Right. Remember it s open source? Open as in: I audit this code as much as I want to? - As in: No response at all after issues were committed to developer. 4/25/14 35
RESPONSES DISCLOSURE SECURITY FIXES Change log or Release notes _never_ mentions security fixes explicitly No hints or information on the developer Web sites! CVE _Common_ never heard about that CREDITS What s that? BUT THERE ARE SOME PROFESSIONALS The Icinga Team has published bug fix releases (incl. back ports), ordered CVE numbers and assign the issues as Security issues. MANY THANKS AND WELL DONE! 4/25/14 36
MITIGATIONS BEST PRACTICES BEST PRACTICES Consider Icinga and Nagios Security Guidelines e.g. http://docs.icinga.org/latest/en/security.html Nothing similar available for Cacti and Check_MK GENERAL BASICS Patching and regular updates OS and middleware hardening Minimal rights on application level, but also on operating system level Remove critical features (e.g. WATO in Check_MK) Passwords 4/25/14 37
MITIGATIONS SEGREGATION ON NETWORK LEVEL Do not place such systems flat in your corporate network Consider segregation based on functions, e.g. own monitoring systems for dedicated services No internet for the admin workstations and monitoring system (incl. ICMP, DNS, NTP, ) ON APPLICATION LEVEL Segregate user and roles 4/25/14 38
MITIGATIONS ARCHITECTURE AGENT BASED MONITORING Needs privileged rights to get all information and listen to the network (often unauthenticated) Security of agents should be discussed separately e.g. http://www.securityfocus.com/archive/1/531063/30/0/threaded CHECK VIA SSH Must be secured carefully via SSHd configuration otherwise direct shell login SOLUTION Change the communication direction Based on Check_MK s agent, it s just a configuration no additional software needed 4/25/14 39
MITIGATIONS ARCHITECTURE HOW IT WORKS Run Check_MK agent locally and pipe output to a file 4/25/14 40
MITIGATIONS ARCHITECTURE HOW IT WORKS Run Check_MK agent locally and pipe output to a file Secure transfer, e.g. via SCP/SFTP 4/25/14 41
MITIGATIONS ARCHITECTURE HOW IT WORKS Run Check_MK agent locally and pipe output to a file Secure transfer, e.g. via SCP/SFTP Configure Check_MK Configuration & Check Engine to get information from a local file 4/25/14 42
MITIGATIONS ARCHITECTURE OWN CHECKS 4/25/14 43
CONCLUSION Take care about your used solutions incl. additional features, add-ons, plug ins, self written checks and architecture. When it named Open Source, it does not mean it is secure itself! In general Open Source Monitoring solutions are not more or less secure than commercial ones. Strong isolation of administrator workstations and your monitoring system as well. @Developer: Check OWASP regularly! 4/25/14 44