OpenTPX v2.2 Oct 8 th 2015 LookingGlass Cyber Solutions PRESENTER:
OpenTPX Contribution OpenTPX is a contribution by LookingGlass Cyber Solutions to the open source community 1 Specifications and source code are distributed under Apache License 2.0 Checkout www.opentpx.org OpenTPX was created to build highly scalable machine-readable threat intelligence, analysis and network security products that exchange data at large volumes and at high speed We welcome your feedback and contributions 1 OpenTPX is designed for optimized network security & threat intelligence use cases and does support mappings for select threat intelligence formats including CSV, STIX, OpenIOC etc. 2
Contents OpenTPX Introduction What Where Who Threat Scores Threat Observables, Associations & Collections Networks, Packet Capture & Mitigation Queries Additional Capabilities 3
Introducing OpenTPX Comprehensive data exchange for the security landscape All context required for Network Security Operations and Threat Intelligence exchange Modular approach Defines threat score model across all elements Designed for efficient data processing Focus on the raw context to convey Minimalist representation of the basic raw observations without significant overhead or confused representations Designed for graph relationships Referencing across multiple data relationships OpenTPX Open source technology sharing spec, code and examples Optimized and extensible data model & representation For machine to machine ingest with large volume and high speed Dictionary keys easily added OpenTPX - Network OpenTPX - Threat OpenTPX - Collections OpenTPX Mitigation OpenTPX Feeds Efficient data ingest designed for highly connected data Easy indexing of data Faster ingest to systems that are typical in threat intelligence Simplified keys identifying types, easy creation Flexibility of schema and data ingest Normalized schema but not limited to extension 4
OpenTPX Scoring & Queries Underpinning OpenTPX building blocks, it provides Comprehensive scoring framework Query Language Scoring Across meta-data, networks, domains, users Query Language Comprehensive language allowing combinatorial queries to be constructed across threat context OpenTPX - Network OpenTPX OpenTPX - Threat OpenTPX - Collections Scoring OpenTPX Mitigation OpenTPX Feeds Query Language 5
OpenTPX Content Categories Threat Observable Dictionary Observable names, their associated criticality, description and the set of classifications to which the observable belongs to The dictionary allows the provider to define observables (e.g meta-data) once and then refer to that observable name for each subject Threat Observable Associations An observable to one or more subjects (i.e. elements) including network, host or user subjects Network subjects include IP, CIDR, ASN, FQDN Host subjects include file hashes, application identifiers, malware identifiers User subjects include user name, user identity, alias, email address Collections may define country information, named grouping of network, host elements and observables A collection may contain zero or more collections Networks where each network may define network membership, routing topology, ownership, network announcements Mitigation What mitigation is recommended for particular threat observable 6
Where OpenTPX is used Trusted Communities and Integrated Systems including Threat Analysts Sends manual defined Collections containing sector or company specific information Malware Analysis Automated Malware Analysis system sends network packet capture and threat observations to Threat Analyst Feed Provider Sends Threat Observables associated with global Internet Threat Intelligence Management System Exchanges all information used to collaborate on security Sharing across systems Network Capture Captures packets and behaviors and sends summarized results on threats Network Security Receives mitigation rules to change security posture Feed Provider Threat Analyst TPX Threat Threat Intelligence Management System TPX Collection TPX - All TPX Network, Threat Threat Intelligence Management System TPX Mitigation TPX Network Malware Analysis Network Capture Network Security 7
Who can benefit from OpenTPX? CERTS/Security Operations To provide information on incidents Threat researchers to exchange all context available that defines a threat not just IOCs but full set of observables including analysis Sensor/feed providers To provide context on network, threats, sectors, actors etc. Security Companies or organizations wishing to exchange common definition of threat segmentation Any Machine to Machine threat exchange Requiring optimal data processing and data exchange for large (Tbytes of data, in real-time) 8
What is a Threat Observable We define the term threat observable loosely to be any observation that may have an associated threat score and may be associated with one or more elements of interest It is deliberate that OpenTPX has a very loose definition of the threat observable to ensure increased flexibility and extensibility Thereby removing some of the rigidity of a more structured approach A threat observable can be one or more of the following: An Indication Of Compromise (IOC) An Originating or Destination Network A network topology A Target Network, domain A Command & Control behavior An application (malware or otherwise) An actor A behavior A TTP A report A human defined note or description Threat observables may be combined into collections and reference each other Threat observables comprise an identifying name, and one or more key/value attributes that capture the observation s data Threat observable attributes keys may come from a pre-defined dictionary or may introduce new terms 9
Threat Scoring OpenTPX
Threat Score Conceptual Model Scoring across the security landscape 1 st layer in the model starts with network devices, topology, routing, endpoints, servers 2 nd layer defines the applications and services that run over the core layer devices 3 rd layer defines the users that run those applications 4 th layer defines the observables and meta-data associated with all of the 3 core layers Observables/ Meta Users Applications Network (infr, hosts) Score
Threat Score Risk scoring across all elements of threat Scores across Observables Course and fine grained Classifications Sources Scores can be associated with both positive and negative observables 12
Threat Sources Individual sources may be scored indicating the provider s confidence "schema_version_s": "2.2.0", "provider_s": "Pcap Intel Provider Company", "list_name_s": "Pcap Provider Company List Data", "source_observable_s": "PCAP_IND_NAME", "source_file_s": "/var/lg/data/json/list_name/2014/06/01/2014.pcap", "source_description_s": "This feed provides information on PCAP behavior captured by X", "distribution_time_t": 1221312312, "last_updated_t": 121232134, "score_i": 90, 13
Threat Classifications Observables can be assigned multiple classifications At least 1 coarse grained classification Each classification has an associated score 14
Threat Classification Scores Individual classifications may be scored "observable_dictionary_c_array": [ { "observable_id_s" : "Conficker A", "criticality_i": 70, score_24hr_decay_i : 40, "description_s": "If an IP address or domain has been associated with this tag, it means that Intel Provider Company has identified the IP address or domain to be associated with the Conficker botnet variant A.", "classification_c_array" : [ { "classification_id_s": "APT", "classification_family_s": "Malware", "score_i": 70 } ], }, 15
Threat Observable Criticality Scores Criticality is how serious or impactful an observation has been assessed by the provider Example "observable_dictionary_c_array": [ { "observable_id_s" : "Conficker A", "criticality_i": 70, score_24hr_decay_i : 40, "description_s": "If an IP address or domain has been associated with this tag, it means that Intel Provider Company has identified the IP address or domain to be associated with the Conficker botnet variant A.", "classification_c_array" : [ { "classification_id_s": "APT", "classification_family_s": "Malware", "score_i": 70 } ], }, 16
Observable Definitions & Associations OpenTPX
Why Observable Dictionary and Association An observable is any network or threat observation An observable has a definition that defines what it represents Defines the identifier, score, description, classification, criticality and common attributes shared across all instances of the observable Observable Definition An observable is then associated with one or more networked assets where that observation has been seen Defines the specific information of the observable as seen on this specific network asset By defining the observable separately from the instance information we avoid duplicative and unnecessary bloat of information focus on the minimum information necessary to convey that observation on a specific IP or Domain Observable Association Observable Association Observable - Asset Association Observable #1 - Asset Association Observable #1 - Asset Association #1 - Asset #1 - Asset #n 18
Observable Dictionary and Association Inheritance Observational Model Information defined in the observable dictionary can be overridden in the instance if necessary Observable Dictionary Define an observable once Acts as the base definition of the observable Can have classification, score, raw behaviors common to all observables of this type Observable Association Associate many times to different subjects The instance of the observable Specific attributes associated with this instance possible allowing for derived definitions { "observable_id_s": Conficker A", "criticality_i": 70, score_24hr_decay_i : 4, "description_s": "If an IP address or domain has been associated with this tag, it means that Intel Provider Company has identified the IP address or domain to be associated with the Clicker botnet.", "classification_c_array": [ { "classification_id_s": "Worm", "classification_family_s": "Malware", "score_i": 70 } "element_observable_c_array": [ { "subject_ipv4_s": "123.123.123.132", "score_i": 90, "threat_observable_c_map": { "Conficker A": { "occurred_at_t": 4355545, "last_seen_t": 13123, "country_code_s": "IR", "destination_fqdn_s": "ddd.com", "score_i": 70, }, 19
Observables over time Efficient observable model allows association rather than repetition of data unnecessarily T0: The dictionary entry is created by the provider. The provider defines the description and the classification of the threat T1: The 1 st instance of the Observable associated with Element #1 The provider observes the Threat associated with an element T1: The 1 st instance of the Observable associated with Element #2 The provider observes the Threat associated with another element T2: The 2 nd instance of the Observable associated with Element #1 The provider observes the Threat again on the same element T0 Threat Observable Dictionary Entry Time T1 Observable Element Association #1 Observable Element Association #2 Element #1 Element #2 T2 Observable Element Association #2 Element #1 20
Threat Intelligence OpenTPX
Threat Intelligence Observable definition in the dictionary Done once Common attributes of this observation shared by all instances "observable_dictionary_c_array": [ { "observable_id_s" : "Conficker A", "criticality_i": 70, "score_i": 72, "summary_s": "This is a summary of the observable", "description_s": "If an IP address or domain has been associated with this tag, it means that Intel Provider Company has identified the IP address or domain to be associated with the Conficker botnet variant A.", "notes_s": "User defined notes", "reference_s_array": [ "http://www.thisisareference.com/observablereference", "http:// www.anotherreference.com/2ndrefererence" ], "classification_c_array" : [ { "classification_id_s": "Malware", "classification_family_s": "Worm", "score_i": 70 } ], }, Observable associated with a subject Done for each subject Subjects can be IP, FQDN, File Hashes etc Specific attributes that define the particular instance with this subject "element_observable_c_array": [ { "subject_ipv4_s": "123.123.123.132", "score_i": 90, "threat_observable_c_map": { "Conficker A": { "occurred_at_t": 4355545, "last_seen_t": 13123, "country_code_s": "IR", "dest_fqdn_s": "ddd.com", "score_i": 70 }, "Clicker": { "occurred_at_t": 4355545, "last_seen_t": 13123, "country_code_s": "CH", "dest_fqdn_s": "aaa.com" } } Distributed }, under Apache License 2.0 http://www.apache.org/licenses/license-2.0 22
Collections OpenTPX
Why Collections? A collection is a group of related entities to convey a structure Use Case #1: Organization assets Problem: Many organizations have multiple CIDRs, Ips, Domains etc that have no direct network linkage but from a security perspective they wish to convey what is important to secure and monitor. Solution: Collections allow an organization to convey a structure to those assets and associate Internet and Threat intelligence with those structures Use Case #2: Industry Segments Problem: Many organizations wish to understand threats associated with industry segments such as financial sector, energy sector etc to understand overall threat health Solution: Collections allow segmentation of organizations and convey threat intelligence across those segments Use Case #3: Incident Investigations Problem: A threat incident may represent a set of networks, malware and other artifacts that need to be conveyed to others working on the incident in a collected form. Solution: Collections allow an incident response team to create the group of information relevant to the incident so that they can share a common view of that information and assess the impact Many other use cases possible 24
Collections Define segments, sectors, user organizations, groups, companies, incidents Collections are hierarchical May have confidence score associated May contains IPs, CIDRs, FQDNs, ASN, Observables, other Collections "collection_c_array": [ { // a top level collection "name_id_s": "MarketSeg1", "last_updated_t": 1212312323, "description_s": "This collection is related to MarketSeg1, "author_s": "Allan Thomson", "workspace_s": "lg-system", // the score of the MarketSeg1 collection "score_i": 90, "collection_c_array": [ { // a 2nd level collection MarketSeg1 -> NCR10205 // with FQDN, IP, CIDR, ASN and sub-collection defined "name_id_s": "NCR10205", "description_s": "This is NCR10205 subcollection", "last_updated_t": 12123132322323, "author_s": "Gerry Eaton", "score_i": 70, "fqdn_c_array": [ { "fqdn_s": "seguintexas.gov" }, { "fqdn_s": "tenaska.com" }, ], "ip_c_array": [ { "ip_ipv4_s": "12.1.1.1" }, { "ip_ipv4_s": "13.1.1.1" }, ], "cidr_c_array": [ { "cidr_cidrv4_s": "208.191.120.72/29" }, { "cidr_cidrv4_s": "208.191.120.64/29" }, ], 25
Networks, Packet Capture & Mitigation OpenTPX
Why Networks? Network information and how the internet is connected represents a fundamental baseline for understanding threats Knowing what networks exist without requiring threat information provides a basis for analysts to understand their exposure and attack surface It also allows them to understand and assess the full scope of networks that are of interest, in the absence of threats Network information contains Topology Upstream connections Downstream connections Advertised routes and sub-networks Ownership 27
Networks Example Useful for describing networks that are involved in threat context Includes: Network topologies Ownership Routers Announcements "asn_c_array": [ { // // This information is for ASN = 1 // "asn_i": 1, "as_owner_s": "ABC Corp", // // The list of routers that are part of the ASN // "asn_routers_ip_array" : [ 1231231, 12312313214, 12131311241, 12312423414 ], // // The router interconnections in the ASN // "asn_router_conns_c_array": [ { "router_1_u": 1231232112, "router_2_u": 121435523 }, { "router_1_u": 2314123434, "router_2_u": 4523423432} ], // // The CIDR announcements from the ASN // "asn_cidr_announcements_c_array": [ { "start_ip_u": 1234567890, "end_ip_u": 2234567890, "aggregator_ip_u": 12332144, "observed_at_t": 213232232 }, { "start_ip_u": 3234567890, "end_ip_u": 4234567890, "aggregator_ip_u": 12332144 }, 28
Packet Capture Captures all packet exchanges Any protocol Any attribute Key/value pairs Optimized data indexing May represent TTPs, Behaviors or patterns "Threat_Inject_tiger_mama": { "dns_request_c_array": [ { "req_fqdn_s": "irc.freenode.net" }, ], "dns_response_c_array": [ { "record_s": "A", "resp_ipv4_s": "12.1.1.1" }, { "record_s": "CNAME", "resp_fqdn_s": "chat.freenode.net"}, { "record_s": "AAAA", "resp_ipv6_s": "2001:708:40:2001:a822:baff:fec4:2428"}, { "record_s": "TXT", "resp_fqdn_s": "google-siteverification=mrswln2ncqsbgduywer9f6y0euau0mr_anpgna0mwes" }], "fqdn_c_array": [ { "fqdn_s": "eff.com", "ip_ipv4_s": "12.1.1.1" }, { "fqdn_s": "isatap.f.sck.im", "ip_ipv4_s": "89.2.11.2"} ], "host_c_array": [ { "host_fqdn_s": "badguy.com" }, ], "http_c_array": [ { "body_s": "", "method_s": "GET", "version_s": "1.1", "agent_s": "Battle.net/1.2.4.5383", "uri_fqdn": "http://us.launcher.battle.net/service/wow/alert/en-us", "dest_port_i": 8653}, { "body_s": "", "method_s": "GET", "version_s": "1.1", "agent_s": "Battle.net/1.2.4.5383", "uri_fqdn": "http://us.launcher.battle.net/service/wow/alert/en-us", "dest_port_i": 8654}, ], "icmp_c_array": [ { "src_ipv4_s": "224.0.0.1", "dest_ipv4_s": "121.1.1.1", "type_i": 9 }, ], 29
Malware Report Captures malware reports including all IOCs without requiring association to specific network assets { } "source_observable_s": "LG CTIG", "list_name_s": "Automated Malware Analysis", "observable_dictionary_c_array": [ { "criticality_i": 60, "classification_c_array": [ { "score_i": 30, "classification_id_s": "Malware Artifacts", "classification_family_s": "Malware" } ], "observable_id_s": "Automated Malware Analysis Report - 0dd3f6a83347768b88f3013dce592d3d", "attribute_c_map": { "magic_s": "PE32 executable (console) Intel 80386, for MS Windows", "tlp_i": 1, "last_seen_t": 1442949180, "dest_fqdn_s_array": [ "xthefo.com", "qyupbu.com", "lbuyzo.com" ], "hash_md5_h": "0dd3f6a83347768b88f3013dce592d3d", "hash_sha256_h": "947875388ff7e99613a58a3af3890d9304d912cb15b388f42875a212035e5f8a", "filepath_s_array": [ "C:\\WINDOWS\\system32\\ntkrnlpa.exe", "\\Device\\NamedPipe\\lsass" ], "registrykey_s_array": [ "HKEY_LOCAL_MACHINE\\SYSTEM\\ControlSet001\\Services\\PDRELI\\ObjectName ], "hash_sha1_h": "a2dbf3a419a0cf9f190b47ae624bf52196617c9c", "filesize_i": 56832, "dest_ipv4_s_array": [ "50.63.202.29", "8.8.8.8", "148.81.111.121" ] }, "description_s": "A report containing the summary of an automated malware detection" } ], "last_updated_t": 1442949180, "score_i": 95, "schema_version_s": "2.2.0", "provider_s": "LookingGlass" 30
Mitigation Supports inheritance mitigation recommendations Enables recommendations at a dictionary level or specific association level of an observable Multiple mitigation terms possible Log Drop Others easily added "threat_observable_c_map": { "Conficker A": { "occurred_at_t": 4355545, "last_seen_t": 13123, "country_code_s": "IR", "destination_fqdn_s": "ddd.com", "score_i": 70, mitigation_c_array : [ { action : log destination 1.1.1.1 }, { action : drop }, ], }, "Clicker": { "occurred_at_t": 4355545, "last_seen_t": 13123, "country_code_s": "CH", "destination_fqdn_s": "aaa.com", mitigation_c_array : [ { action : log 1/1 }, ], } 31
OpenTPX Query Language (QL) OpenTPX
OpenTPX QL Introduction Is a dialect of Solr Lucene with extensions that ease querying network elements OpenTPX QL supports advanced query grouping ranges wildcarding of values passed to terms Allows providers and consumers to exchange queries as part of threat context Examples observable_s:zeus - return all entities where observable_s matches Zeus observable_s:banking* - return all entities where observable_s begins with Banking url_s:http://msn*.com - return all entities where the URL begins with msn and ends with.com. timestamp_i:[1414503194 TO *] - return all entities which were last updated sooner than 1414503194 NOT observable_s:banking* - return all entities where observable_s does NOT begin with Banking (ip_i:10.0.0.1 AND observable_s:banking*) OR (ip_i:10.0.0.2 AND observable_s:trojan*) - return any Banking observable associated with 10.0.0.1 IP address, or return any Trojan observable associated with 10.0.0.2 IP address 33
Language Syntax whitespace = { " " "\t" "\n" "\r" } ; string = '"', { characters }, '"' ; integer = { '0', '1', '2', '3', '4','5', '6', '7', '8', '9' }, field-separator = ':' ; group-begin = '(', [ whitespace ] ; group-end = ')', [ whitespace ] ; range-begin = '[', [ whitespace ] ; range-end = [ whitespace ], ']' ; range-to = whitespace, 'TO', whitespace ; wildcard = '*' ; wildcard_single = '?' ; and-token = [ whitespace ], 'AND', [ whitespace ] ; or-token = [ whitespace ], 'OR', [ whitespace ] ; not-token = [ whitespace ], 'NOT' '!', [ whitespace ] ; symbol = [ whitespace symbol ] begin-of-input, {characters}, [ whitespace end-of-input ]; range = range-begin, [ integer, string, symbol, wildcard ], whitespace, range-to, whitespace, [ integer, string, symbol, wildcard ]; term = symbol, field-separator, [ string, symbol, integer, range ] ; and = { term, group, and, or, not }, and-token, { term, group, and, or, not }; or = { term, group, and, or, not }, or-token, { term, group, and, or, not }; not = not-token, { term, group, and, or, not } ; group = group-begin, { group, term, and, or, not }, group-end ; 34
OpenTPX QL Basic Queries The most basic Query in OpenTPX QL is a single Term Terms are in the following format: field:value which is a Field, followed by a :, followed by a Value. Examples: foo:bar - searches foo for String "bar". foo:5 - searches foo for Integer 5. All Fields in queries against a data store is typed explicitly as either an Integer or String. The Field in a query is typed by appending the relevant type indicator to the Field: Integer - _i String - _s 35
OpenTPX QL Wildcard Queries In OpenTPX Solr, the following wildcards are supported in Values: * - Wildcard - multi-character? - WildcardChar - single-character Wildcards have the following restrictions: Wildcards are not permitted for Integers Values cannot start with a wildcard. i.e. left-anchored wildcards in strings such as *foobar will not be accepted Values can themselves be a single Wildcard to express a query that wishes to select for the existence of a Field Fields cannot contain wildcards Examples: observable_s:banking* - return all entities where observable_s begins with Banking. sha1_s:????f4f4e4cf2f9669cc61e2565effcd8f923d28 - return all entities where the last 36 characters of the sha1_s match the provided hex digest. url_s:http://msn*.com - return all entities where the URL begins with msn and ends with.com. 36
OpenTPX QL - Grouping Supports sub query grouping, which can be useful for altering the order and precedence of the Boolean statements. Groups are begun with the (character and terminated with the) character. Groups can also be nested to an arbitrary depth, as needed. Example: Default: a:1 b:2 OR c:3 would evaluate as an implicit AND in between a:1 and b:2. OR takes precedence before AND in Boolean Algebra, so this would evaluate as (a:1 AND (b:2 OR c: 3)). Example: (ip_i:10.0.0.1 AND observable_s:banking*) OR (ip_i:10.0.0.2 AND observable_s:trojan*) 37
Additional Capabilities OpenTPX
OpenTPX Dictionary and Extensions OpenTPX specifies dictionary of terms used for many common protocols, networks, threat observables Examples: occurred_at_i New terms easily added without pre-registration New OpenTPX terms require no registration to be added Contributors are encouraged to add common terms they consider to be missing back to the community New OpenTPX Observables require no registration to be distributed via OpenTPX files 39
OpenTPX Structural Options Ingest of OpenTPX content is intended to be efficient and focused on machine-to-machine communications Option #1: Single Payload/File Ideal for smaller payloads Containing just one feed option threat observations Collections Networks Mitigations Option #2: Manifest + Multiple Payload/Files Ideal for larger payloads Containing a manifest file that indexes other content in separate files No limit to number of files 40
Protocols to exchange OpenTPX data OpenTPX content may be transported by any transport protocol that makes sense for a machine to machine exchange Examples in use: Syslog SMTP HTTP Rsync FTP 41