Big Data Architectures: Concerns and Strategies for Cyber Security

Transcription

1 Big Data Architectures: Concerns and Strategies for Cyber Security David Blockow Software Architect, Data to Decisions CRC au.linkedin.com/in/davidblockow Executive summary. Big Data architectures and advanced analytics have the potential to provide cyber security capability beyond that which is available in traditional tools. The techniques used to analyse high volume and velocity data from varied sources can present unique challenges for assurance and risk management. This paper discusses the concerns arising from the use of Big Data architectures for cyber security, including capturing and presenting provenance metadata for machine-generated assertions and the difficulties securing Big Data infrastructure effectively. A number of design patterns and strategies are presented to address the concerns, and areas requiring further research to more adequately address concerns are identified. Introduction Big Data tools and techniques are being increasingly applied to the cyber security domain, for example, to detect spam 1 or to detect similarity in malware 2. Advanced data analytics can also support an effective defence against cyber intrusions, providing near realtime actionable security alerts, reducing the time from detection to remediation and enabling pro-active defence strategies. Applying Big Data tools to intrusion detection can be challenging due to the characteristics of the data sources, including very high volume and velocity sources (e.g. network packet captures) and a large variety of sources with variable quality (e.g. operating system and application logs). Traditional tools typically analyse live data but do not store and analyse historical data due to the computational and storage overhead. Big Data tools and techniques can store and leverage the historical data, providing context for new data and improving the accuracy of data analytics. Additionally, the accuracy of data analytics can often be improved by combining multiple data sources or by simply 1 Machine Learning Techniques in Spam Filtering, Konstaintin Tretyakov, University of Tartu 2 A Similarity-Based Machine Learning Approach for Detecting Adversarial Android increasing the amount of data available for analysis 3. The Data to Decisions Cooperative Research Centre 4 has developed a Big Data Reference Architecture (BDRA) to assist solution designers with architecting effective Big Data applications. The BDRA identifies a number of architectural concerns that a solution designer should consider, and presents strategies that can be used to address each concerns. This paper draws on the content of the BDRA, discussing the concerns and strategies in the context of a cyber security use case performing network packet anomaly detection. Relevant concerns and strategies are presented and shortfalls in current capability are identified. Sample Solution: Network Anomaly Detection A sample anomaly detection solution is presented as an example of applying Big Data tools and Machine Learning (ML) techniques to the cyber security domain. This is a sample pipeline and does not represent a real-world system, but serves as a useful tool to ground Malware, Doaa Hassan, Matthew Might, Vivek Srikumar, University of Utah 3 The Unreasonable Effectiveness of Data, Alon Halevy, Peter Norvig, Fernando Pereira, Google 4 1

2 Figure 1 - Network Packet Anomaly Detection the discussion. The high-level architecture is presented in Figure 1 - Network Packet Anomaly Detection. Network packets, operating system logs and application logs are collected in an ingest buffer, then forked between a scale-out NoSQL cluster and a stream processing framework. The raw data is periodically processed by batch analytics to train anomaly detector and anomaly classification machine learning models. The anomaly detector model learns normal characteristics of the network and generates output when unusual behaviour is detected. The anomaly classification model can classify the type of behaviour (e.g. an intrusion attempt, an exfiltration attempt) based on previously observed and labelled anomalies. Unclassified anomalies can be output as an unknown event for investigation. Periodic re-training of the models is necessary to adapt to changes in network behaviour. The stream processing analytics execute the anomaly detector model over a sliding window of live data. Detected anomalies are classified using the anomaly classification model and a near real-time alert is generated for a user (and/or an automated system) to act upon. A relational database stores the machine learning models and the detected anomalies for review. A user interface allows the user to review anomalies and the evidence supporting them as well as labelling unclassified anomalies to support model training. Utilising historical data allows the models to learn normal behaviour over a long period of time. For example, an event that results in periodic increased outbound traffic (e.g. an infrequently scheduled offsite backup) could incorrectly be detected as unusual if analysed in isolation, but would not be detected as unusual by a model that considers historical data. A high-level workflow for alert generation is illustrated in Figure 2 - Anomaly Detection Workflow. Raw data undergoes an enrichment and feature extraction process after ingest; enrichment adds additional information to the raw data e.g. geo-locating IP addresses, while feature extraction parses and extracts features from the raw data for use in the ML models e.g. packet sizes, log levels. This pipeline is representative of a simple real-world analytic pipeline much longer and complex pipelines are common. 2

3 Figure 2 - Anomaly Detection Workflow The BDRA identifies many architectural concerns across a broad mix of use cases and domains. The two most relevant areas discussed in this paper are Provenance and Security, with particular emphasis on the assurance and risk management aspects. Provenance Provenance in this example refers to the audit trail or chain of evidence that lead to a machine assertion. Provenance metadata for the sample architecture could include: The data sources that were ingested, including estimates of their quality The time range of raw data that was analysed to generate the alert The type of enrichment and feature extraction processes that were applied The version of machine learning models that generated the alert and classified the anomaly The training data that was used to train the models The users that labelled the training data There is a risk with Big Data solutions that machine assertions will be treated as truth, when in reality many machine learning outputs are probabilistic. Provenance metadata can increase the quality of decisions, either by supporting user interfaces that present the likelihood along with an assertion, or by providing an analyst with the ability to discover and review the data behind the assertion. Provenance Concerns Capturing and presenting provenance for Big Data applications can be challenging for a number of reasons, discussed below. Diverse Data Sets As previously discussed, Big Data analytics can benefit from combining and analyzing multiple data sets. This can complicate provenance metadata as multiple inputs must be recorded, with varying data source characteristics and quality. To assist an analyst in making quality decisions the provenance metadata should capture enough information for them to understand the characteristics of each data source and the implications of data source quality on the analytic process. Polyglot Persistence Polyglot persistence refers to the use of multiple storage technologies in a Big Data solution, for example, in the sample architecture data is stored across both a NoSQL store and a relational database. A related concept is data de-normalisation, where multiple representations of the data are stored within the same data store to support different query patterns. Provenance metadata should be maintained for all representations of the data across a system. Advanced Analytic Pipelines Big Data solutions often utilise non-trivial analytic pipelines. Provenance metadata for a long analytic pipeline will not be human readable. Additionally, metadata that can be captured for a machine learning model is limited; often only the version of the model is recorded. In the sample architecture we would know when the anomaly detector model detects an anomaly, but we can t easily explain why it detected the anomaly. Provenance Metadata Management For systems recording detailed provenance information the size of the metadata can be 3

4 larger than data itself. Metadata will often have a different structure or schema compared to the raw or processed data, and may require the use of different storage technologies. Provenance Strategies While provenance metadata management can prove challenging in the Big Data domain, a number of strategies and design patterns can be applied to address the provenance concerns. Data Source Catalog A data source catalog captures metadata for each source, including human-readable notes describing the characteristics and quality of the data. The information captured in a data catalogue can support decision making when combined with provenance metadata. For the sample architecture a data catalog would include the sources of network packet captures and a list of log file sources, potentially with notes on the quality of log output for each application. Data Preparation Data preparation is a common processing step performed after ingesting data into a Big Data system. Preparation performs validation and cleansing to remove or correct erroneous records, and transforms the data to support later steps in the analytic pipeline. Data preparation can improve the quality of decisions by reducing errors in source data and normalising the quality of input data to some extent. Changes made to the data during preparation should be recorded with the provenance metadata. Provenance Data Management Provenance data management techniques treat provenance metadata as a first-class data source, defining metadata schemas, managing changes to schemas over time (e.g. by versioning the schema), defining storage technologies (often a dedicated triplestore) and controlling access to the metadata separately to the data (often with more stringent access controls). Some effort has been made to standardise on provenance schemas, for example, PROV- DM 5. These models typically define a schema that captures the action that was performed (e.g. cleansing), the actor that performed the action (e.g. a username or the version of an analytic) and the target of the action (e.g. a data set or record). Provenance/Confidence Visualisation Provenance metadata for any non-trivial system will not be human readable. This can be addressed by converting the metadata into a confidence or likelihood indicator for the user. For some machine learning techniques a probability is a natural output, for others the probability may need to be calculated using other ad-hoc mechanisms. For the sample architecture confidence values could be combined with techniques such as threat modelling to rank alerts in order of priority. A sample representation of this is shown in Table 1 - Confidence Visualisation. Table 1 - Confidence Visualisation Time Alert Conf. 04:39:58 Data Exfiltration 95% 03:22:12 Brute Force Auth 62% 03:44:34 Port Scan 23% Security The Security section of the BDRA addresses protection of the infrastructure from compromise. This is particularly important for the sample application as a compromised system could be modified to mask the behaviour it is designed to detect. Traditional security controls should be applied to protect Big Data systems, however there are characteristics of Big Data architectures that require additional security strategies. Security Concerns High Value Targets Big Data solutions typically combine multiple organisational data sources in a single location, 5 4

5 often using multiple technologies and products in a solution architecture. These characteristics combine to produce a high-value target with a large attack surface. Product Heritage Many Big Data technologies, particularly NoSQL databases, were developed to solve a single use case, then generalised and open sourced 6. This heritage produces products that are not generally designed with security in mind; security is often an afterthought, and default configurations are typically insecure. Diverse Tools Big Data architectures using multiple products will often need to reconcile approaches to security, with varying levels of maturity and support for security controls across different products. Applying access controls consistently across a Big Data system can be challenging. Security Strategies Secure Clusters Where possible, NoSQL clusters should be configured to authenticate cluster membership, and inter-cluster network traffic should be protected with SSL/TLS. These controls are available in some NoSQL products, however they are not usually enabled by default as they can complicate cluster administration. Perimeter-Based Security for Clusters Perimeter-based approaches to security are no longer considered best practice, however the shortfalls of in NoSQL cluster security mean they are often used in practice. For clusters that do not provide fine-grained authentication or the ability to encrypt internal traffic the cluster can be firewalled from the rest of the organisation, with trusted edge nodes used to interact with the data. For the sample architecture this would involve a firewall and edge node for communicating with the NoSQL cluster. Consistent Access Controls Big Data system components will ideally be integrated with a shared identity provider to apply access controls consistently across products. Support for identity providers varies amongst products, which often necessitates the use of role mapping techniques where product interfaces are adapted with an API shim or façade that maps between different roles and access control mechanisms. For the sample architecture this could involve an API shim across the relational and NoSQL stores that applies access controls consistently. Data-Driven Intrusion Detection Intrusion detection is one area where Big Data solutions can have an advantage over traditional IT systems. Big Data systems can take a data-driven approach to intrusion detection, monitoring the infrastructure using the storage and processing capabilities that are already available in the architecture. Research Areas The strategies presented above will mitigate some of the architectural concerns. Areas requiring further research and improvement include: Provenance for Machine Learning Ongoing research aims to understand the inner working of common machine learning techniques such as neural networks 7. This research can help answer why a machine learning model generated an output, however the current state of the art provides only lowlevel details that are not useful for someone unfamiliar with the algorithms. A tool that can present (at a higher level) the significant 6 Big Data: Architectures and Technologies, John Klein, CMU Software Engineering Institute 7 Understanding Neural Networks Through Deep Visualization, Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, Hod Lipson 5

6 features of the source data that lead to a machine learning output would be beneficial. metadata and identifying common security shortfalls in distributed systems. Effective Provenance Visualisation Complex provenance metadata generated from non-trivial analytic pipelines is difficult for a human to interpret. Current approaches to consolidate the metadata into a confidence value or threat score are often ad-hoc. Systems could benefit from a robust and/or standard approach for determining and visualising confidence. Securing Big Data Products While strategies exist to secure big data infrastructure, they are mostly Band-Aid solutions to address shortfalls in the underlying products. Research could be undertaken to identify common pitfalls in distributed system product design, providing a set of recommendations and guidance for open source projects. Conclusion This paper has identified architectural concerns and strategies for a sample cyber security use case performing network anomaly detection, focussing on Provenance and Security concerns that affect assurance and risk management. Provenance concerns can be addressed through the use of a data source catalog capturing quality information, data preparation for cleansing and validation, provenance metadata management for capturing a detailed audit trail and provenance visualisation for presenting confidence values to end users. Big Data specific security concerns can be addressed through securing NoSQL clusters using both internal and perimeter-based approaches, applying access controls consistently through the use of API facades, and taking a data driven approach to intrusion detection. Areas requiring further research include calculating and presenting provenance for machine learning techniques, effectively calculating and visualising provenance 6