Accelera'ng Your Solu'on Development with Splunk Reference Apps

Transcription

1 Copyright 2015 Splunk Inc. Accelera'ng Your Solu'on Development with Splunk Reference Apps Grigori Melnik Principal Product Manager Developer PlaAorm,

2 Disclaimer During the course of this presentagon, we may make forward looking statements regarding future events or the expected performance of the company. We caugon you that such statements reflect our current expectagons and esgmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentagon are being made as of the Gme and date of its live presentagon. If reviewed aser its live presentagon, this presentagon may not contain current or accurate informagon. We do not assume any obligagon to update any forward looking statements we may make. In addigon, any informagon about our roadmap outlines our general product direcgon and is subject to change at any Gme without nogce. It is for informagonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligagon either to develop the features or funcgonality described or to include any such feature or funcgonality in a future release. 2

3 How should I build my app?

4 Splunk Developer Guidance EVERYTHING YOU NEED TO BUILD

5 Building SoluGons on the Splunk PlaAorm Splunk Reference Apps Complete, working real- world Splunk solugons built together with partners (Conducive, Auth0) Splunk Developer Guide This is unbelievable, it covers most everything I learned the hard way Bernie Macias, Technical Architect, Zillow dev.splunk.com/goto/devguide

6 Planning a journey PlaDorm and tools: a kitbag for our journey UI and visualiza'ons: what the apps look like Working with data: where it comes from & how we manage it Adding code: using JavaScript and Search Processing Language Packaging and deployment: reaching our first desgnagon Dealing with OAuth Aler'ng Building in telemetry with high- performance data collec'on splk.it/devguide

7 Splunk Reference App Demo

8 How we build guidance

9 1. Started with a QuesGons Backlog! Architecture Why should I not use transacgons? What does a typical Splunk applicagon reference architecture look like? When should I use pivot vs tstats? What common paradigms are applicable to Splunk app development? Why should I use data models? What are the typical deployment topologies? Why should I choose a specific one? What are the confounding factors on the choice of my topology? When my data source touches on many data models, should I assume complete separagon or heavy inheritance? How do I extend an exisgng data model? How do I parggon my Splunk solugons? What does CIM offer and why should I build CIM- compliant apps? What are the tradeoffs of various types of inputs? In the context of CIM, what are the tradeoffs of using my props.conf and transforms.conf and rewrigng them on How do I architect my Splunk solugon and deployment for a very large scale? indexing, completely discarding the vendor supplied field names? How do I reconcile the advantages of a clean How do I architect my Splunk solugon for the cloud? What are specific consideragons for deploying to AWS or Azure? interface & normalisagon, but at the cost of losing alignment with published vendor documentagon, and a learning What s the landscape of Splunk extension points? curve for exisgng users? How do I integrate data from Splunk into exisgng applicagons and systems? How do I manage my solugon declaragve configuragon? How do I detect/troubleshoot bad config? How do I plan and design a robust alergng and monitoring subsystem on top of Splunk? How do I log and analyze data that is not event driven (certain web feeds, html parsing, image meta data)? Compare and contrast ad- hoc searching vs background searching What should I consider for my sizing requirements? What are recommended configuragons of Splunk deployment to meet my sizing requirements? How do I handle transient faults? Should I architect my solugon to index my data in local data center (zone) or centrally? How do I effecgvely manage credengals? What are things we can automagcally degrade so we can make sure our core experience is working? What s the effect of search head locagon on my app and the overall user experience? How When something happens, how effecgvely propagate the info and react to it? How do I develop an integrated mechanism to let me connect Splunk to my MOM (messaging middleware) and index my messages? How are other What solugons on Splunk does do I prepare built? What were a the challenges? typical How have they Splunk been addressed? applica'on architecture be produc've look like? do I integrate my event Splunk genera'on into exis'ng when systems? developing & How do I package an app? deal with versioning and updates? How do I handle the requirement that app configs must be different across different server types in a distributed! Packaging and Deployment How do I piece together various parts of a Splunk app (custom search commands, mod inputs etc.)? How do I package a Splunk solugon with a single install that automagcally rolls out all the necessary dependencies? How do I manage my Splunk solugon versioning, backward and future compat? What's the best way to split up custom apps for deployment?! Development How should I set up my development environment to be producgve with Splunk? What are different ways of how I develop my Splunk app? Pros and cons of using specific SDK vs REST APIs? Pros and cons of using SimpleXML vs Advanced XML vs Web Framework How do I analyze a data source for a TA? What are the different ways of enriching the data in Splunk? What are their tradeoffs? When should I use event types and transacgons for data classificagon? How do I extend Splunk to define a custom input capability? When should I use modular inputs vs scripted inputs vs..? What are streaming vs non- streaming outputs consideragons? How do I deal with long- running scripts? Handling shutdown/restart of Splunk? Concurrency? State persistence etc. tes'ng Splunk? an app? environment (e.g. apps on search heads shouldn't have inputs enabled)?! Quality/Compliance What quality gates should I consider? What kind of para- funcgonal characterisgcs are important to consider? What heurisgcs do I use to bless/block a release? How do I test a data model? How do I prepare event generagon when building/tesgng an app? What kind of perf tesgng should I do and how? How do I test UI? How do I security cergfy my solugon? How do I design to sagsfy my retengon and compliance policies? How do I architect to design my availability requirements? How do I handle geographic disaster recovery / fault tolerance? How do I properly instrument my solugon so that I know what s happening?! Sustained Engineering How do I maintain/service/support Splunk apps? How do my customers handle updagng their customized configs once new versions of my app come out?! Business Why should I build on Splunk? 9 What kind of skill do I need my devs to have to build a Splunk solugon? What is the community building? How are current devs creagng unique experiences using Splunk I typically want to

10 2. IdenGfied Extensibility Surface Area Data inges'on & indexing Input ê Scripted inputs ê Modular inputs ê Custom (trained) source types ê Custom sources Data inges'on pipeline ê Field extracgons ê Field transformagons Indexing ê Custom indexes Searching Search authoring ê Custom search commands ê Macros (basic, parametrized) ê Saved searches Data classifica'on ê Event types ê TransacGons Data enrichment ê Lookups ê KV store collecgons ê Workflow acgons Data normaliza'on ê Tags ê Aliases Data mining ê cluster & dedup ê anomalousvalue ê kmeans ê predict commands Processing & repor'ng Search- 'me mapping ê Data models CIM extensions Custom UI/visualiza'ons ê Pages, views & dashboards ê JS Extensions ê CSS Extensions ê Custom setup screens Scheduled processing ê Scheduled reports Aler'ng ê Scripted alerts ê Custom alert acgons Branding & naviga'on ê Custom app navigagon & branding Manageability ê Custom splunkweb controllers ê Custom splunkd endpoints 10

11 3. Mined business requirements with partner 4. Formulated learning objecgves 5. Reconciled 3 & 4 with our designs 11

12 Data Search language AggregaGng siloed metrics into meaningful KPIs Data manipulagon Data normalizagon Sub- searches Config- driven Persistence with KV store Macros Viz: Dynamic scaling Customizing in- the box viz controls Custom nav Ux acgviges permeagng all dev Using sub- searches to correlate data TroubleshooGng searches Data mining: ExploraGon PreparaGon: filtering/deduping/ buckegng Using advanced stagsgcs funcgons Threshold- based anomaly detecgon EvaluaGng goodness /accuracy General search paqerns Search opgmizagons Ux Prototyping AdapGng 3 rd party viz library Composite charts with interacgons Dealing with high- volume data sets TroubleshooGng perf issues Post- process or not- post- process deployment implicagons Automated UI tesgng (w.selenium) Post- processing IntegraGng with 3 rd party component Unit tesgng (w.mocha) PersisGng state (per user) Plus non- funcgonal topics: App versioning Packaging InstallaGon Security review Deployment Publishing to splunkbase App cergficagon Data modeling Using lookups Building a baseline lookup table Windows of Gme/Custom Gme ranges Overlaying Gme data Serng the stage Overall Splunk app structure UI technology selecgon: Simple XML vs SplunkJS Modularity Dev & test env Dev workflow Modularity Data onboarding CIM compliance Tools

13 Takeaways! PlaDorm, not just an engine! On- prem and cloud! App development!= rocket science! Splunk Developer Guidance : learn and reuse for the win!! Reach out to my team (devinfo@splunk.com) and tell us about your experience

14 Resources dev.splunk.com/goto/devguide github.com/splunk/splunk- ref- pas- code github.com/splunk/splunk- ref- pas- 14

15 Related breakout sessions & acgviges A Lap Around Developer Awesomeness in the Next Version of Splunk (Itay Neeman/Glenn Block) Visualizing Data From the Ground Up: Raw Data to Interac've Graphics With Splunk (Marshall Agnew) Liberate Your Applica'on Logging! (Glenn Block/Jian Lee) Modular Inputs - If You Build It, They Will Come (ScoA Haskell) The 'State' of Splunk - Using the KVStore to Maintain App State (Stefan Sievert) Advanced Interac'ons Using SimpleXML (Mathew ElFng, Siegfried Puchbauer) Splunk Apps You Can't Miss, and the People Who Make Them (Hal RoAenberg) 15

16 THANK YOU