A Big Data Platform for Developers

Transcription

2 Developer Evangelist at Splunk since July 2012 Splunk Community Member Splunk for JMX SplunkJavaLogging SplunkBase Apps and Answers Splunk Architect and Administrator Coder Been paying my mortgage developing Enterprise Java solutions most of my career Kia Ora About me I do not have a speech impediment, I am from Aotearoa, so please restrain all your sheep, Lord of the Rings and Kim Dotcom heckles until beer o clock!! 2

3 Overview of the Splunk platform Splunk for Developers Custom Visualization Demo Splunk Java SDK Spring Integration Splunk Extensions Integration Adaptors Demo Some other JVM/Java related tools SplunkJavaLogging Splunk for JMX Questions Agenda 3

4 What is

5 So What is Splunk, Exactly? Splunk is an engine for machine data Provides visibility, reporting and search across all your IT systems and infrastructure Doesn t lock you into a fixed schema It s software download and install it in 5 minutes, freemium model Runs on all modern platforms Open and extensible architecture 5

6 Indexes any Machine Data Capture events from logs in real time Run scripts to gather system metrics, connect to APIs and databases Listen to syslog, raw TCP/UDP, gather Windows events Universally indexes any data format so it doesn t need adapters, schema on the fly Stream in data directly from your application code Decode binary data and feed in Windows Registry Event logs File system sysinternals Linux/Unix Configurations Syslog File system Ps, iostat, top Virtualization Hypervisor Guest OS Guest Apps Applications Web logs Log4J, JMS, JMX.NET events Code and scripts Databases Configurations Audit/query logs Tables Schemas Network Configurations syslog SNMP netflow 6

7 Centralizes Data Across the Environment Splunk Universal Forwarder sends data to Splunk Indexer from remote systems Uses minimal system resources, easy to install and deploy Delivers secure, distributed, real-time universal data collection for tens of thousands of endpoints Indexing/Search Server Splunk Forwarders 7

8 Scales to TBs/day and Thousands of Users Automatic load balancing linearly scales indexing Distributed search and MapReduce linearly scales search and reporting 8

9 Provides Strong Machine Data Governance Provides comprehensive controls for data security, retention and integrity Single sign-on integration enables pass-through authentication of user credentials 9

10 Splunk is an implementation of the Map Reduce algorithmic approach It is not Apache Hadoop MapReduce(MR) the product Splunk is not agnostic of its underlying data source, optimized to Splunk Index files Real time vs Batch Jobs Optimal for time series based data End to End Integrated Big Data Solution Fine grained protection of access and data using role based permissions Data retention and aging controls Users can submit Map Reduce jobs without needing to know how to code a job Splunk Search Language vs Pig/Sawzill But why not get the best of both worlds 10 Splunk and Apache Hadoop MR/HDFS Splunk Hadoop Ops Splunk Hadoop Connect Shuttl (archiving to HDFS / S3)

11 Splunk Has Four Primary Functions Searching and Reporting (Search Head) Indexing and Search Services (Indexer) Local and Distributed Management (Deployment Server) Data Collection and Forwarding (Forwarder) A Splunk install can be one or all roles 11

12 Getting Data into Splunk Agent and Agent-less Approach for Flexibility. syslog TCP/UDP Local File Monitoring log files, config files dumps and trace files syslog compa>ble hosts and network devices Mounted File Systems WMI \\hostname\mount Event Logs Performance Unix, Linux and Windows hosts Ac>ve Directory shell code perf Scripted Inputs shell scripts custom parsers batch loading virtual host Windows Inputs Event Logs performance counters registry monitoring AcAve Directory monitoring Agent- less Data Input Windows hosts Custom apps and scripted API connec>ons Splunk Forwarder Windows hosts 12

13 Delivers secure, distributed, real-time universal data collection for 10 s of thousands of endpoints Extends Splunk data fabric to large scale private cloud and desktop environments Uses minimal system resources, easy to install and deploy < half memory and footprint of Splunk 4.1; <1% of single core Universal Data Forwarder Forward data without negahvely impachng produchon performance. Logs Universal Forwarder Deployment Messages ConfiguraHons Metrics Central Deployment Management Scripts Monitor files, changes and the system registry; capture metrics and status. 13

14 Horizontal Scaling Load balanced search and indexing for massive, linear scale out. Distributed Search Forwarder Auto Load Balancing 14

15 Multiple Datacenters Index and store locally. Distribute searches to datacenters, networks & geographies. Distributed Search Headquarters London Hong Kong Tokyo New York 15

16 Send Data to Other Systems Route raw data in real time or send alerts based on searches. Service Desk Event Console Problem InvesHgaHon SIEM

17 High Availability / DR Combine auto load balancing and data replication. Distributed Search Primary Cluster Data Clone Secondary Cluster Splunk Forwarders Auto Load Balancing 17

18 Integrate External Data Extend search with lookups to external data sources. LDAP, AD Watch Lists CMDB CRM/ ERP Correlate IP addresses with locahons, accounts with regions 18

19 Integrate Users and Roles Integrate authentication with LDAP and Active Directory. LDAP, AD Users and Groups Splunk Flexible Roles CapabiliHes & Filters Manage Indexes Share Searches Problem InvesHgaHon Problem InvesHgaHon Problem InvesHgaHon Manage Users Save Searches NOT tag=pci App=ERP Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter. 19

20 Centralized Licensing Management Groups, Stacks, and Pools for Enterprise Deployments. Problem InvesHgaHon 20

21 Deployment Monitoring Keep Tabs On Your Splunk Enterprise Deployment. Licenses Sourcetypes Indexers Forwarders 21

22 Real-time Search Data Monitor Input TCP/UDP Input Scripted Input Parsing Queue Parsing Pipeline Source, event typing Character set normalizahon Line breaking Timestamp idenhficahon Regex transforms Index Queue Real- Hme Buffer Indexing Pipeline Raw data Index Files Real- Hme Search Process Index 22

23 Real-time Alerting source= /var/log/secure.log BAD SU Data Monitor Input TCP/UDP Input Scripted Input Parsing Queue Parsing Pipeline Source, event typing Character set normalizahon Line breaking Timestamp idenhficahon Index Queue Real- Hme Buffer Regex transforms Indexing Pipeline Raw data Index Files Real- Hme Search Process Index 23

24 New Approach to Heterogeneous Data Universal Indexing Search-time Knowledge Flexibility and Fast Time to Value No data normalization Automatically handles timestamps Parsers not required Index every term & pattern blindly No attempt to understand up front Knowledge applied at search-time No brittle schema to work around Multiple views into the same data Splunk helps find transactions, patterns and trends Normalization as it s needed Faster implementation Easy search language Multiple views into the same data 24

25 Inside Universal Indexing AutomaHc event boundary idenhficahon AutomaHc Hmestamp normalizahon...enable accurate searching and trending by Hme across all data: 25

26 Inside Search-time Knowledge Extraction AutomaHcally discovered fields And user- defined fields... enable stahshcs and precise search on specific fields: 26

27 Inside Search-time Knowledge Extraction Searches saved as event types Plus tagging of event types, hosts and other fields... enable normalized reporhng, knowledge sharing and granular access control. 27

28 28 Splunk for Developers

29 Splunk & Developers Machine Data SplunkUI (Splunk Apps) REST API Custom/ Existing Applications SDKs Search, chart and graph Save and schedule searches as alerts Export search results Manage inputs and indexes Add & remove users and roles Engine Accelerate development & testing Integrate data from Splunk into your existing IT environment for operational visibility Build custom solutions to deliver real-time business insights from Big Data 29

30 Splunk in the Developer Community Over 1,000 unique visitors per week to dev.splunk.com Over 500 followers on Over 350 enterprise developer trial licenses granted

31 Accelerate development & testing

32 How does Splunk Accelerate Dev/Test? Splunk frees you from upfront database design for analytics late binding schema Developers and QA/test engineers don t have to ask IT/Ops to get logs off machines Role base access to all data within one console without having to log into production systems All events are indexed and accessible in real-time in one place. Ad-Hoc real-time monitoring and historical investigation searchable from one place Correlations and insights across multiple tiers. Splunk lets you find issues quickly, so you can fix issues quickly Integrate Splunk search results into testing assertions 32

33 Splunk filled a vacuum we didn t know we had. - Nathan Pratt, Tech Lead, Tools & Automation, StubHub Started with Site Operations to resolve issues Grew to engineers, QA, upper management in technology Release requirement Projects are required to certify that all logs are Splunk-friendly StubHub & Splunk Engineering uses Splunk to investigate bugs QA uses it during dev cycles 33 High-level view of application errors - used by site operations, engineering, and upper management

34 Integrate Splunk into your IT environment

35 Integration into existing IT tools The Splunk development platform is optimized for core enterprise developer skills Splunk UI (Splunk Apps) REST API splunkd Your application SDKs REST API communicates directly with a Splunk instance for search, management and admin Provides full control to the developer Use any language or tool that supports HTTP SDKs provide broad coverage of the REST API in popular languages Log directly to Splunk from any app Build a UI on any web stack Integrate into existing infrastructure 35

36 Splunk REST API Exposes an API method for every feature in the product Whatever you can do in the UI you can do through the API. Run searches Manage Splunk configurations API is RESTful Endpoints are served by splunkd Requests are GET, POST, and DELETE HTTP methods Responses are Atom XML Feeds JSON coming in 5.0 Search results can be output in CSV/JSON/XML/Raw 36

37 We want to make it as easy as possible for developers to build Big Data apps on top of the Splunk platform Several different language offerings, Software Development Kits (SDKs) Javascript, Java, Python, PHP, C#(private), Ruby(private) All Splunk functionality is accessible via our SDKs Get Data into Splunk Execute Splunk Searches, get data out of Splunk Manage Splunk Customized User Interfaces Developer Platform SDKs 37

38 Comcast & Splunk Content browsed, purchased and watched All tracked by time and MAC address + Customer profile and MAC address / device assignments Correlate usage and profile data to analyze customer behavior: Revenues driven by content browsed Improving local content mix Better search results Tailor content promotion 38

39 Splunking data sent from ARM-based devices Uses the Java SDK to send data to Splunk Bosch & Splunk Evidencebased Telehealth Healthcare Management Cardiac Rhythm Monitoring 39

40 Splunk as an integrated, enterprise-ready Big Data platform

41 Splunk = Integrated, Enterprise- ready Big Data Plajorm No need to write MapReduce jobs, just get data into Splunk and analyze Splunk delivers real-time insight like clickstream analysis, IT early-warning systems, security and fraud protection Late-binding schema allows for faster, more flexible data insight gathering Data collection is integrated Distributed architecture offers scale-out capabilities with access control Out-of-the-box reporting and analytics capabilities SDKs cover over 170 REST API endpoints 41

42 Splunk eliminates the need to write large MapReduce jobs to get meaningful information out of our data. This means we can get powerful stats and information to our key stakeholders in a fraction of the time. - Isaac Mosquera, CTO, Socialize Socialize & Splunk 42

43 Visualizing Splunk with the SDKs Splunkweb has rich, but sometimes limited, visualization options You can use the SDKs to extract data from Splunk using a search, and visualize it Real-time searches can be especially powerful Using the Javascript SDK you can integrate with third party charting librarys like Google Charts & D3. 43

44

45 Realtime Twitter Visualization Demo Twitter feeds being firehosed into Splunk and searched over in realtime Uses the Splunk Javascript SDK to stream the realtime search results from Splunk into a totally customized web based user interface Visualization of most popular hashtags with interactive pie chart,word cloud and geo heatmap using D3 Javascript SDK Browser 45

46 46 Realtime Twitter Demo

47 47 Splunk Java SDK(Software Development Kit)

48 Open sourced under the Apache v2.0 license Clone from Github : git clone Project level support for Eclipse and Intellij IDE s Pre-requisites JRE 6+ Ant ( Maven support is in the works ) Splunk installed Loads of code examples Project examples folder Unit Tests Comprehensive coverage of the REST API Get the Java SDK 48

49 Java SDK Class Model HTTPService Resource Service ResourceCollection Entity EntityCollection Application Index Input InputCollection SavedSearchCollection Collections use a common mechanism to create and remove entities Entities use a common mechanism to retrieve and update property values, and access entity metadata Service is a wrapper that facilitates access to all Splunk REST endpoints 49

50 Connect and Authenticate Manage Input Events Search Key Java SDK Use cases 50

51 Connect and Authenticate public static Service connectandlogintosplunkexample() { Map<String, Object> connectionargs = new HashMap<String, Object>(); connectionargs.put("host", somehost"); connectionargs.put("username", spring"); connectionargs.put("password", integration"); connectionargs.put("port", 8089); connectionargs.put("scheme", "https"); // will login and save the session key which gets put in the HTTP Authorization header Service splunkservice = Service.connect(connectionArgs); return splunkservice; } 51

52 Manage public static void getserverinfoexample() { } Service splunkservice = connectandlogintosplunkexample(); ServiceInfo info = splunkservice.getinfo(); System.out.println("Info:"); for (String key : info.keyset()) System.out.println(" " + key + ": " + info.get(key)); Entity settings = splunkservice.getsettings(); System.out.println("\nSettings:"); for (String key : settings.keyset()) System.out.println(" " + key + ": " + settings.get(key)); 52

53 public static void logeventtosplunkexample() { } Input Events Service splunkservice = connectandlogintosplunkexample(); // Get a Receiver object Receiver receiver = splunkservice.getreceiver(); // Set the sourcetype Args logargs = new Args(); logargs.put("source", http-rest"); logargs.put("sourcetype", spring-example"); // Log an event into the spring index receiver.log( spring", logargs, SpringOne 2GX rocks"); Other Input transports HTTP REST Streaming Raw TCP Oneshot & Streaming Raw UDP & Syslog 53

54 54 Search Search query a set of commands and functions you use to retrieve events from an index or a real-time stream, "search index=spring error OR exception head 10 Saved search a search query that has been saved to be used again and can be set up to run on a regular schedule Search job an instance of a completed or still-running search operation.using a search ID you can access the results of the search when they become available. Job results are saved for a period of time on the server and can be retrieved Search Modes Normal : asynchronous, poll job for status and results Realtime : same as normal, but stream is kept open a results streamed in realtime Blocking : synchronous, a job handle is returned when search is completed Oneshot : synchronous, no job handle is returned, results are streamed Export : synchronous, not a search per say, doesn t create a job, results are streamed oldest to newest

55 public static void exportsearchexample() { Blocking Searches Service splunkservice = connectandlogintosplunkexample(); String searchquery = "search error OR exception head 10"; Args queryargs = new Args(); queryargs.put("earliest_time", "-1d@d"); queryargs.put("latest_time", "now"); // perform the export, blocks here InputStream stream = splunkservice.export(searchquery, queryargs); processinputstream(stream); } public static void simplesearchexample() { Service splunkservice = connectandlogintosplunkexample(); String searchquery = "search error OR exception head 10"; Args queryargs = new Args(); queryargs.put("earliest_time", "-3d@d"); queryargs.put("latest_time", "-1d@d"); // perform the search, blocks here InputStream stream = splunkservice.search(searchquery, queryargs); processinputstream(stream); } 55

56 public static void searchjobexample() { } Non Blocking Search Service splunkservice = connectandlogintosplunkexample(); String outputmode = "csv";// xml,json,csv // submit the job Job job = splunkservice.getjobs().create("search index=spring error OR fatal head 10"); while (!job.isdone()) { try {Thread.sleep(500);} catch (Exception e) {} } Args outputargs = new Args(); outputargs.put("output_mode", outputmode); InputStream stream = job.getresults(outputargs); processinputstream(stream, outputmode); // uses xml stream, opencsv and gson 56

57 public static void realtimesearchexample() { Realtime Search Service splunkservice = connectandlogintosplunkexample(); Args queryargs = new Args(); queryargs.put("earliest_time", "rt-5m"); queryargs.put("latest_time", "rt"); // submit the job Job job = splunkservice.getjobs().create("search index=spring exception OR error, queryargs); } 57

58 Alternate JVM Languages Scala Groovy Clojure Javascript(Rhino) JRuby PHP(Quercus) Ceylon Kotlin Jython We don t need SDK s for these languages, we can just use the Java SDK! 58

59 Groovy class SplunkJavaSDKWrapper { } static main(args) { //connect and login def connectionparameters = [host: somehost",username:"spring",password:"integration"] Service service = Service.connect(connectionParameters) //get Splunk Server info ServiceInfo info = service.getinfo() def splunkinfo = [:] for (key in info.keyset()) splunkinfo.put(key,info.get(key)) printsplunkinfo(splunkinfo) } static printsplunkinfo(splunkinfo) { println "Info splunkinfo.each { key, value ->println key + " : " + value} } 59

60 Scala import com.splunk.service._ import scala.collection.mutable.hashmap import scala.collection.javaconversions._ object SplunkJavaSDKWrapper { } def main(args: Array[String]) = { } //connect and login val connectionargs = HashMap[String, Object]("host" -> somehost,"username" -> me,"password" -> foo") val service = connect(connectionargs) //get Splunk Server info val info = service.getinfo // Scala/Java conversion val javaset = info.keyset val scalaset = javaset.toset //print out Splunk Server info for (key <- scalaset) println(key + ":" + info.get(key)) 60

61 Spring Integration Splunk Extensions Special thanks to Jianwei Li(Jarred) & Mark Pollack for creating this! 61

62 Spring Integration Spring Integration is an extension to core Spring Based on Enterprise Integration Patterns model Messaging model and Declarative Adaptors Makes it easier to build integration solutions 62

63 Spring Integration Splunk Adaptors Splunk Java SDK makes it easier to use the REST API Building on this, the Spring Integration Adaptors make it easier for Spring/Java developers to declaratively build data integration solutions and utilize the power of the Splunk platform Inbound Adaptor Search and export the data from Splunk and push into message channels Filter, transform, export to other destinations Outbound Adaptor Can consume data acquired by other Integration adaptors(twitter, JDBC ) and push it into Splunk for indexing, searching and visualization 63

64 Spring Integration Splunk Inbound Adaptor Blocking, Non Blocking, Saved & Realtime Searches Exporting 64

65 Spring Integration Splunk Outbound Adaptor HTTP REST Input TCP Input 65

66 Common Splunk settings 66 XML Configuration <int-splunk:server id="splunkserver" host= somehost" port="8089" username= damien" password= foobar"/> Searching/exporting from Splunk <int-splunk:inbound-channel-adapter id="splunkinboundchanneladapter auto-startup="true" search="search index=spring error OR exception splunk-server-ref="splunkserver channel="inputfromsplunk" mode="blocking" initearliesttime="-1d"> <int:poller fixed-rate="5" time-unit="seconds"/> </int-splunk:inbound-channel-adapter> Inputting events to Splunk <int-splunk:outbound-channel-adapter id="splunkoutboundchanneladapter" auto-startup="true" order="1 channel="outputtosplunkwithmessagestore" splunk-server-ref="splunkserver pool-server-connection="true" index="spring" sourcetype="twitter-feed" source="spring-integration-httprest ingest="submit"> </int-splunk:outbound-channel-adapter>

67 67 Spring Integration Splunk Twitter Demo

68 68 SplunkJavaLogging

69 A logging framework to allow developers to as seamlessly as possible integrate Splunk best practice logging semantics into their code and transport events directly to Splunk. Custom handler/appender implementations(rest and Raw TCP) for the 3 most prevalent Java logging frameworks in play. Splunk events directly from your code. LogBack Log4j java.util.logging Better handling of stacktraces SplunkJavaLogging All code and examples is on Github 69

70 70 Splunk for JMX

71 SplunkBase App for monitoring JVM Applications Out of the box dashboards for JVM level monitoring (java.lang domain) Memory, Threading, GC, CPU etc Very simple configuration to wire up monitoring of any Mbeans from applications (Tomcat, Jboss, Cassandra, Coherence etc ) Hotspot, JRockit, IBMJ9, OpenJDK Poll JMX attributes and operations, index data over time, correlate with other data Supports large scale deployments of JVMs Extensible and Customizable Many connectivity options RMI, IIOP Direct Process Attachment Splunk for JMX MX4J Hessian, Burlap and Soap Freely available download from SplunkBase & all code is on Github 71

72 Learn More. Stay Connected. At SpringOne 2GX : Come by our booth Splunk demos,q & A SDK code Tee Shirts!! Web : Developer Platform : SplunkBase : [email protected], [email protected] Blog : Github : Splunk Live! Events and Online Videos at 72

73 73 Thanks for coming.