A Big Data Platform for Developers Damien Dallimore Developer Evangelist at Splunk 2012 SpringOne 2GX. All rights reserved. Do not distribute without permission.
Developer Evangelist at Splunk since July 2012 Splunk Community Member Splunk for JMX SplunkJavaLogging SplunkBase Apps and Answers Splunk Architect and Administrator Coder Been paying my mortgage developing Enterprise Java solutions most of my career Kia Ora About me I do not have a speech impediment, I am from Aotearoa, so please restrain all your sheep, Lord of the Rings and Kim Dotcom heckles until beer o clock!! 2
Overview of the Splunk platform Splunk for Developers Custom Visualization Demo Splunk Java SDK Spring Integration Splunk Extensions Integration Adaptors Demo Some other JVM/Java related tools SplunkJavaLogging Splunk for JMX Questions Agenda 3
What is
So What is Splunk, Exactly? Splunk is an engine for machine data Provides visibility, reporting and search across all your IT systems and infrastructure Doesn t lock you into a fixed schema It s software download and install it in 5 minutes, freemium model Runs on all modern platforms Open and extensible architecture 5
Indexes any Machine Data Capture events from logs in real time Run scripts to gather system metrics, connect to APIs and databases Listen to syslog, raw TCP/UDP, gather Windows events Universally indexes any data format so it doesn t need adapters, schema on the fly Stream in data directly from your application code Decode binary data and feed in Windows Registry Event logs File system sysinternals Linux/Unix Configurations Syslog File system Ps, iostat, top Virtualization Hypervisor Guest OS Guest Apps Applications Web logs Log4J, JMS, JMX.NET events Code and scripts Databases Configurations Audit/query logs Tables Schemas Network Configurations syslog SNMP netflow 6
Centralizes Data Across the Environment Splunk Universal Forwarder sends data to Splunk Indexer from remote systems Uses minimal system resources, easy to install and deploy Delivers secure, distributed, real-time universal data collection for tens of thousands of endpoints Indexing/Search Server Splunk Forwarders 7
Scales to TBs/day and Thousands of Users Automatic load balancing linearly scales indexing Distributed search and MapReduce linearly scales search and reporting 8
Provides Strong Machine Data Governance Provides comprehensive controls for data security, retention and integrity Single sign-on integration enables pass-through authentication of user credentials 9
Splunk is an implementation of the Map Reduce algorithmic approach It is not Apache Hadoop MapReduce(MR) the product Splunk is not agnostic of its underlying data source, optimized to Splunk Index files Real time vs Batch Jobs Optimal for time series based data End to End Integrated Big Data Solution Fine grained protection of access and data using role based permissions Data retention and aging controls Users can submit Map Reduce jobs without needing to know how to code a job Splunk Search Language vs Pig/Sawzill But why not get the best of both worlds 10 Splunk and Apache Hadoop MR/HDFS Splunk Hadoop Ops Splunk Hadoop Connect Shuttl (archiving to HDFS / S3)
Splunk Has Four Primary Functions Searching and Reporting (Search Head) Indexing and Search Services (Indexer) Local and Distributed Management (Deployment Server) Data Collection and Forwarding (Forwarder) A Splunk install can be one or all roles 11
Getting Data into Splunk Agent and Agent-less Approach for Flexibility. syslog TCP/UDP Local File Monitoring log files, config files dumps and trace files syslog compa>ble hosts and network devices Mounted File Systems WMI \\hostname\mount Event Logs Performance Unix, Linux and Windows hosts Ac>ve Directory shell code perf Scripted Inputs shell scripts custom parsers batch loading virtual host Windows Inputs Event Logs performance counters registry monitoring AcAve Directory monitoring Agent- less Data Input Windows hosts Custom apps and scripted API connec>ons Splunk Forwarder Windows hosts 12
Delivers secure, distributed, real-time universal data collection for 10 s of thousands of endpoints Extends Splunk data fabric to large scale private cloud and desktop environments Uses minimal system resources, easy to install and deploy < half memory and footprint of Splunk 4.1; <1% of single core Universal Data Forwarder Forward data without negahvely impachng produchon performance. Logs Universal Forwarder Deployment Messages ConfiguraHons Metrics Central Deployment Management Scripts Monitor files, changes and the system registry; capture metrics and status. 13
Horizontal Scaling Load balanced search and indexing for massive, linear scale out. Distributed Search Forwarder Auto Load Balancing 14
Multiple Datacenters Index and store locally. Distribute searches to datacenters, networks & geographies. Distributed Search Headquarters London Hong Kong Tokyo New York 15
Send Data to Other Systems Route raw data in real time or send alerts based on searches. Service Desk Event Console Problem InvesHgaHon SIEM
High Availability / DR Combine auto load balancing and data replication. Distributed Search Primary Cluster Data Clone Secondary Cluster Splunk Forwarders Auto Load Balancing 17
Integrate External Data Extend search with lookups to external data sources. LDAP, AD Watch Lists CMDB CRM/ ERP Correlate IP addresses with locahons, accounts with regions 18
Integrate Users and Roles Integrate authentication with LDAP and Active Directory. LDAP, AD Users and Groups Splunk Flexible Roles CapabiliHes & Filters Manage Indexes Share Searches Problem InvesHgaHon Problem InvesHgaHon Problem InvesHgaHon Manage Users Save Searches NOT tag=pci App=ERP Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter. 19
Centralized Licensing Management Groups, Stacks, and Pools for Enterprise Deployments. Problem InvesHgaHon 20
Deployment Monitoring Keep Tabs On Your Splunk Enterprise Deployment. Licenses Sourcetypes Indexers Forwarders 21
Real-time Search Data Monitor Input TCP/UDP Input Scripted Input Parsing Queue Parsing Pipeline Source, event typing Character set normalizahon Line breaking Timestamp idenhficahon Regex transforms Index Queue Real- Hme Buffer Indexing Pipeline Raw data Index Files Real- Hme Search Process Index 22
Real-time Alerting source= /var/log/secure.log BAD SU Data Monitor Input TCP/UDP Input Scripted Input Parsing Queue Parsing Pipeline Source, event typing Character set normalizahon Line breaking Timestamp idenhficahon Index Queue Real- Hme Buffer Regex transforms Indexing Pipeline Raw data Index Files Real- Hme Search Process Index 23
New Approach to Heterogeneous Data Universal Indexing Search-time Knowledge Flexibility and Fast Time to Value No data normalization Automatically handles timestamps Parsers not required Index every term & pattern blindly No attempt to understand up front Knowledge applied at search-time No brittle schema to work around Multiple views into the same data Splunk helps find transactions, patterns and trends Normalization as it s needed Faster implementation Easy search language Multiple views into the same data 24
Inside Universal Indexing AutomaHc event boundary idenhficahon AutomaHc Hmestamp normalizahon...enable accurate searching and trending by Hme across all data: 25
Inside Search-time Knowledge Extraction AutomaHcally discovered fields And user- defined fields... enable stahshcs and precise search on specific fields: 26
Inside Search-time Knowledge Extraction Searches saved as event types Plus tagging of event types, hosts and other fields... enable normalized reporhng, knowledge sharing and granular access control. 27
28 Splunk for Developers
Splunk & Developers Machine Data SplunkUI (Splunk Apps) REST API Custom/ Existing Applications SDKs Search, chart and graph Save and schedule searches as alerts Export search results Manage inputs and indexes Add & remove users and roles Engine Accelerate development & testing Integrate data from Splunk into your existing IT environment for operational visibility Build custom solutions to deliver real-time business insights from Big Data 29
Splunk in the Developer Community Over 1,000 unique visitors per week to dev.splunk.com Over 500 followers on Twitter @splunkdev Over 350 enterprise developer trial licenses granted
Accelerate development & testing
How does Splunk Accelerate Dev/Test? Splunk frees you from upfront database design for analytics late binding schema Developers and QA/test engineers don t have to ask IT/Ops to get logs off machines Role base access to all data within one console without having to log into production systems All events are indexed and accessible in real-time in one place. Ad-Hoc real-time monitoring and historical investigation searchable from one place Correlations and insights across multiple tiers. Splunk lets you find issues quickly, so you can fix issues quickly Integrate Splunk search results into testing assertions 32
Splunk filled a vacuum we didn t know we had. - Nathan Pratt, Tech Lead, Tools & Automation, StubHub Started with Site Operations to resolve issues Grew to engineers, QA, upper management in technology Release requirement Projects are required to certify that all logs are Splunk-friendly StubHub & Splunk Engineering uses Splunk to investigate bugs QA uses it during dev cycles 33 High-level view of application errors - used by site operations, engineering, and upper management
Integrate Splunk into your IT environment
Integration into existing IT tools The Splunk development platform is optimized for core enterprise developer skills Splunk UI (Splunk Apps) REST API splunkd Your application SDKs REST API communicates directly with a Splunk instance for search, management and admin Provides full control to the developer Use any language or tool that supports HTTP SDKs provide broad coverage of the REST API in popular languages Log directly to Splunk from any app Build a UI on any web stack Integrate into existing infrastructure 35
Splunk REST API Exposes an API method for every feature in the product Whatever you can do in the UI you can do through the API. Run searches Manage Splunk configurations API is RESTful Endpoints are served by splunkd Requests are GET, POST, and DELETE HTTP methods Responses are Atom XML Feeds JSON coming in 5.0 Search results can be output in CSV/JSON/XML/Raw 36
We want to make it as easy as possible for developers to build Big Data apps on top of the Splunk platform Several different language offerings, Software Development Kits (SDKs) Javascript, Java, Python, PHP, C#(private), Ruby(private) All Splunk functionality is accessible via our SDKs Get Data into Splunk Execute Splunk Searches, get data out of Splunk Manage Splunk Customized User Interfaces Developer Platform SDKs 37
Comcast & Splunk Content browsed, purchased and watched All tracked by time and MAC address + Customer profile and MAC address / device assignments Correlate usage and profile data to analyze customer behavior: Revenues driven by content browsed Improving local content mix Better search results Tailor content promotion 38
Splunking data sent from ARM-based devices Uses the Java SDK to send data to Splunk Bosch & Splunk Evidencebased Telehealth Healthcare Management Cardiac Rhythm Monitoring 39
Splunk as an integrated, enterprise-ready Big Data platform
Splunk = Integrated, Enterprise- ready Big Data Plajorm No need to write MapReduce jobs, just get data into Splunk and analyze Splunk delivers real-time insight like clickstream analysis, IT early-warning systems, security and fraud protection Late-binding schema allows for faster, more flexible data insight gathering Data collection is integrated Distributed architecture offers scale-out capabilities with access control Out-of-the-box reporting and analytics capabilities SDKs cover over 170 REST API endpoints 41
Splunk eliminates the need to write large MapReduce jobs to get meaningful information out of our data. This means we can get powerful stats and information to our key stakeholders in a fraction of the time. - Isaac Mosquera, CTO, Socialize Socialize & Splunk 42
Visualizing Splunk with the SDKs Splunkweb has rich, but sometimes limited, visualization options You can use the SDKs to extract data from Splunk using a search, and visualize it Real-time searches can be especially powerful Using the Javascript SDK you can integrate with third party charting librarys like Google Charts & D3. 43
Realtime Twitter Visualization Demo Twitter feeds being firehosed into Splunk and searched over in realtime Uses the Splunk Javascript SDK to stream the realtime search results from Splunk into a totally customized web based user interface Visualization of most popular hashtags with interactive pie chart,word cloud and geo heatmap using D3 Javascript SDK Browser 45
46 Realtime Twitter Demo
47 Splunk Java SDK(Software Development Kit)
Open sourced under the Apache v2.0 license Clone from Github : git clone https://github.com/splunk/splunk-sdk-java.git Project level support for Eclipse and Intellij IDE s Pre-requisites JRE 6+ Ant ( Maven support is in the works ) Splunk installed Loads of code examples Project examples folder Unit Tests http://dev.splunk.com http://gist.github.com/damiendallimore Comprehensive coverage of the REST API Get the Java SDK 48
Java SDK Class Model HTTPService Resource Service ResourceCollection Entity EntityCollection Application Index Input InputCollection SavedSearchCollection Collections use a common mechanism to create and remove entities Entities use a common mechanism to retrieve and update property values, and access entity metadata Service is a wrapper that facilitates access to all Splunk REST endpoints 49
Connect and Authenticate Manage Input Events Search Key Java SDK Use cases 50
Connect and Authenticate public static Service connectandlogintosplunkexample() { Map<String, Object> connectionargs = new HashMap<String, Object>(); connectionargs.put("host", somehost"); connectionargs.put("username", spring"); connectionargs.put("password", integration"); connectionargs.put("port", 8089); connectionargs.put("scheme", "https"); // will login and save the session key which gets put in the HTTP Authorization header Service splunkservice = Service.connect(connectionArgs); return splunkservice; } 51
Manage public static void getserverinfoexample() { } Service splunkservice = connectandlogintosplunkexample(); ServiceInfo info = splunkservice.getinfo(); System.out.println("Info:"); for (String key : info.keyset()) System.out.println(" " + key + ": " + info.get(key)); Entity settings = splunkservice.getsettings(); System.out.println("\nSettings:"); for (String key : settings.keyset()) System.out.println(" " + key + ": " + settings.get(key)); 52
public static void logeventtosplunkexample() { } Input Events Service splunkservice = connectandlogintosplunkexample(); // Get a Receiver object Receiver receiver = splunkservice.getreceiver(); // Set the sourcetype Args logargs = new Args(); logargs.put("source", http-rest"); logargs.put("sourcetype", spring-example"); // Log an event into the spring index receiver.log( spring", logargs, SpringOne 2GX rocks"); Other Input transports HTTP REST Streaming Raw TCP Oneshot & Streaming Raw UDP & Syslog 53
54 Search Search query a set of commands and functions you use to retrieve events from an index or a real-time stream, "search index=spring error OR exception head 10 Saved search a search query that has been saved to be used again and can be set up to run on a regular schedule Search job an instance of a completed or still-running search operation.using a search ID you can access the results of the search when they become available. Job results are saved for a period of time on the server and can be retrieved Search Modes Normal : asynchronous, poll job for status and results Realtime : same as normal, but stream is kept open a results streamed in realtime Blocking : synchronous, a job handle is returned when search is completed Oneshot : synchronous, no job handle is returned, results are streamed Export : synchronous, not a search per say, doesn t create a job, results are streamed oldest to newest
public static void exportsearchexample() { Blocking Searches Service splunkservice = connectandlogintosplunkexample(); String searchquery = "search error OR exception head 10"; Args queryargs = new Args(); queryargs.put("earliest_time", "-1d@d"); queryargs.put("latest_time", "now"); // perform the export, blocks here InputStream stream = splunkservice.export(searchquery, queryargs); processinputstream(stream); } public static void simplesearchexample() { Service splunkservice = connectandlogintosplunkexample(); String searchquery = "search error OR exception head 10"; Args queryargs = new Args(); queryargs.put("earliest_time", "-3d@d"); queryargs.put("latest_time", "-1d@d"); // perform the search, blocks here InputStream stream = splunkservice.search(searchquery, queryargs); processinputstream(stream); } 55
public static void searchjobexample() { } Non Blocking Search Service splunkservice = connectandlogintosplunkexample(); String outputmode = "csv";// xml,json,csv // submit the job Job job = splunkservice.getjobs().create("search index=spring error OR fatal head 10"); while (!job.isdone()) { try {Thread.sleep(500);} catch (Exception e) {} } Args outputargs = new Args(); outputargs.put("output_mode", outputmode); InputStream stream = job.getresults(outputargs); processinputstream(stream, outputmode); // uses xml stream, opencsv and gson 56
public static void realtimesearchexample() { Realtime Search Service splunkservice = connectandlogintosplunkexample(); Args queryargs = new Args(); queryargs.put("earliest_time", "rt-5m"); queryargs.put("latest_time", "rt"); // submit the job Job job = splunkservice.getjobs().create("search index=spring exception OR error, queryargs); } 57
Alternate JVM Languages Scala Groovy Clojure Javascript(Rhino) JRuby PHP(Quercus) Ceylon Kotlin Jython We don t need SDK s for these languages, we can just use the Java SDK! 58
Groovy class SplunkJavaSDKWrapper { } static main(args) { //connect and login def connectionparameters = [host: somehost",username:"spring",password:"integration"] Service service = Service.connect(connectionParameters) //get Splunk Server info ServiceInfo info = service.getinfo() def splunkinfo = [:] for (key in info.keyset()) splunkinfo.put(key,info.get(key)) printsplunkinfo(splunkinfo) } static printsplunkinfo(splunkinfo) { println "Info splunkinfo.each { key, value ->println key + " : " + value} } 59
Scala import com.splunk.service._ import scala.collection.mutable.hashmap import scala.collection.javaconversions._ object SplunkJavaSDKWrapper { } def main(args: Array[String]) = { } //connect and login val connectionargs = HashMap[String, Object]("host" -> somehost,"username" -> me,"password" -> foo") val service = connect(connectionargs) //get Splunk Server info val info = service.getinfo // Scala/Java conversion val javaset = info.keyset val scalaset = javaset.toset //print out Splunk Server info for (key <- scalaset) println(key + ":" + info.get(key)) 60
Spring Integration Splunk Extensions Special thanks to Jianwei Li(Jarred) & Mark Pollack for creating this! 61
Spring Integration Spring Integration is an extension to core Spring Based on Enterprise Integration Patterns model Messaging model and Declarative Adaptors Makes it easier to build integration solutions 62
Spring Integration Splunk Adaptors Splunk Java SDK makes it easier to use the REST API Building on this, the Spring Integration Adaptors make it easier for Spring/Java developers to declaratively build data integration solutions and utilize the power of the Splunk platform https://github.com/springsource/spring-integration-extensions Inbound Adaptor Search and export the data from Splunk and push into message channels Filter, transform, export to other destinations Outbound Adaptor Can consume data acquired by other Integration adaptors(twitter, JDBC ) and push it into Splunk for indexing, searching and visualization 63
Spring Integration Splunk Inbound Adaptor Blocking, Non Blocking, Saved & Realtime Searches Exporting 64
Spring Integration Splunk Outbound Adaptor HTTP REST Input TCP Input 65
Common Splunk settings 66 XML Configuration <int-splunk:server id="splunkserver" host= somehost" port="8089" username= damien" password= foobar"/> Searching/exporting from Splunk <int-splunk:inbound-channel-adapter id="splunkinboundchanneladapter auto-startup="true" search="search index=spring error OR exception splunk-server-ref="splunkserver channel="inputfromsplunk" mode="blocking" initearliesttime="-1d"> <int:poller fixed-rate="5" time-unit="seconds"/> </int-splunk:inbound-channel-adapter> Inputting events to Splunk <int-splunk:outbound-channel-adapter id="splunkoutboundchanneladapter" auto-startup="true" order="1 channel="outputtosplunkwithmessagestore" splunk-server-ref="splunkserver pool-server-connection="true" index="spring" sourcetype="twitter-feed" source="spring-integration-httprest ingest="submit"> </int-splunk:outbound-channel-adapter>
67 Spring Integration Splunk Twitter Demo
68 SplunkJavaLogging
A logging framework to allow developers to as seamlessly as possible integrate Splunk best practice logging semantics into their code and transport events directly to Splunk. Custom handler/appender implementations(rest and Raw TCP) for the 3 most prevalent Java logging frameworks in play. Splunk events directly from your code. LogBack Log4j java.util.logging Better handling of stacktraces SplunkJavaLogging All code and examples is on Github 69
70 Splunk for JMX
SplunkBase App for monitoring JVM Applications Out of the box dashboards for JVM level monitoring (java.lang domain) Memory, Threading, GC, CPU etc Very simple configuration to wire up monitoring of any Mbeans from applications (Tomcat, Jboss, Cassandra, Coherence etc ) Hotspot, JRockit, IBMJ9, OpenJDK Poll JMX attributes and operations, index data over time, correlate with other data Supports large scale deployments of JVMs Extensible and Customizable Many connectivity options RMI, IIOP Direct Process Attachment Splunk for JMX MX4J Hessian, Burlap and Soap Freely available download from SplunkBase & all code is on Github 71
Learn More. Stay Connected. At SpringOne 2GX : Come by our booth Splunk demos,q & A SDK code Tee Shirts!! Web : Developer Platform : http://dev.splunk.com SplunkBase : http://splunk-base.splunk.com Twitter : @splunkdev, @damiendallimore Email : devinfo@splunk.com, ddallimore@splunk.com Blog : http://blogs.splunk.com/dev Github : http://github.com/splunk Splunk Live! Events and Online Videos at http://www.splunk.com 72
73 Thanks for coming.