Conducting a Successful Cloudmarket CIO

Transcription

1 As companies emerge from challenging economic times and turn the corner to face tremendous opportunities, the CIO's role is more and more about strategy and optimizing business results. Today s complex IT solutions create enormous quantities of system data and logs that are rarely analyzed or understood. Hidden within this data are critical clues to customer habits, business behavior, security events, and indications of system health, performance and capacity constraints. The challenge, as well as the enormous opportunity, is to mine this data and harvest substantial business benefits to your company s advantage. 1

2 I'm excited about IT moreso than any time in history and it just keeps getting better think of what the future holds for business and technology as we merge the best of consumer and business revolutions. Cloud services have gone from CRM centric case, to an overall enterprise application, and infrastructure play. This democratization of technology is bringing unbelievable opportunity for innovation and productivity With our business customers more involved than ever in the selection, rollout and maintenance of our enterprise applications, what is the role of the CIO? Applications: 2

3 Cloud Business Applications Hosted Exchange Intranet (2011) Splunk for Operational Intelligence Infrastructure: Hub Data Center (Equinix) Cloud Infrastrcture (Amazon, Skytap) Global Telephony (Shoretel) Antivirus (Symantec) Directory Services (AD/SSO 2011) 2

4 CIO can and has meant many things over time besides Chief Information Officer I argue that today, the most important I in CIO is intelligence In fact I would propose that there are three main objectives for the new Cloudmarket economy CIO 3

5 1) MANAGE TECHNOLOGY INVESTMENTS 2) PROTECT INFORMATION ASSETS 3) IMPROVE BUSINESS OUTCOMES You need to bring more insight into your services, operations, customers, partners, ecosystem, than your customers and lines of businesses can alone Harness the power of transactional business intelligence, along with mountains of machine data to bring operational intelligence 4

6 Don t want to sound too new age but several key transformations are needed to focus CIO s to a new delivery model Need to go from reporting on historical business transactions to ensuring customer experience in real time. Need to go from answering how did the business do last month, to how well does the business run? Our sales channels, our customer support, or web presence 5

7 We CIO s are sitting on a major opportunity to bring incredible opportunity to our businesses by harnessing machine data which in volume is being described as big data Big Data just means: datasets that grow so large that they become awkward to work with using onhand database management tools. Difficulties include capture, storage,search, sharing, analytics, and visualizing. The majority of this data will be unstructured complex data poorly suited to management by structured storage systems like relational databases. The bigness is clearly a technical challenge no doubt, but really not that interesting unless you like cool algorithms and lots of hardware. Data Volume 6

8 Data Diversity Time from question to answer Time from event to insight Time from insight to action Ability to share the data Ability to visualize the data According to IDC, the amount digital information produced in 2011 will be ten times that produced in 2006: 1,800 exabytes 6

9 So much value, so little time. 7

10 Not long ago, the typical data analysis scenario would go something like: Gather requirements ( what questions will we want to answer) Hire expensive data warehouse dba s and scientist Pay for expensive database/warehouse technology Implement schema Implement ETL Test Eventually deploy Then 9 months later you realize the questions have changed 8

11 Over time, IT has developed as silos of systems, focused on specific technologies, functions, departments, groups of systems and people, etc. As a consequence, IT ends up being managed as silos, with narrow, focused tools that provided a limited view of what s really going on. What s more, all IT systems in these silos generate data. This machine generated IT data or exhaust contains a categorical record of behavior behavior of customers, user transactions, networks, servers, applications, and more. This data helps diagnose and fix issues, but is also a source of critical intelligence for the business. Even purported, Single Panes of Glass, like SIEMs, Application Performance Management, Event Correlation and Analysis systems and Data Warehouses, don t provide the complete picture, because they aren t designed for the full scope of this data. Today s IT management tools, security solutions and even business intelligence systems are NOT designed to leverage the full scope of machine generated IT data. Data which is non standard, unstructured, high volume and generated every millisecond of every day.

12 10

13 The common theme with logs is that they lack standards. Let s look at logs generated by custom applications. Applications run in environments that are complex made up of diverse technologies used over time client/server, revamped legacy, packaged applications, Java EE and.net, SOAbased, etc. Most homegrown and packaged applications write local logfiles, often via logging services built into middleware J2EE application servers like Weblogic, Websphere and JBoss,.Net, PHP and more. These files are critical for day to day debugging of production applications by developers and application support. They're also often the best way to report on business and user activity and detect fraud scenarios, since they have all the details of transactions. When developers put timing information into their log events, they can also be used to monitor and report on application performance. However, log messages are unpredictable and highly variable even when coming from a single source. Custom applications can run new code, which result in entirely new log events at any time. Application log files often span multiple lines, an issue for most traditional log management solutions. There is little to no visibility of applications running within virtualized environments or the cloud and access restrictions to production systems impact the ability of Tier 1 help desk staff or developers to diagnose and fix problems. This is a significant challenge for most traditional management tools, which rely on an upfront schema, building custom parsers for known vendor formats, instead of supporting the dynamic nature of application logs. Splunk started first with this custom application logging challenge, designing an approach which indexes all IT data, regardless of format or location. Splunk doesn t rely on traditional RDBMS technology, so there are no schemas which break as soon as new log formats are introduced. And unlike the majority of other log vendors, Splunk can also handle multi line logs.

14 The lack of standards transcends all log sources, from custom applications to logs from packaged applications, servers and network devices. There is literally an A Z list of vendors each generating a multitude of formats, even for the same device. Timestamp formats all over the map, entries are structurally different, the data itself is different! Traditional approaches to log management require custom parsers to translate this plethora of formats, types and sources, because they have been built and designed for a more static world of IT. Splunk indexes all your IT data and getting data into Splunk is easy. You can: Stream data to Splunk over any TCP or UDP network port Read live files on any mounted file system Batch upload files to a spool directory on your Splunk server Run Splunk as a lightweight forwarder to collect many different sources of data and forward them over a single, secure network port to one or more Splunk servers

15 Logs contain the record of activity for IT components, such as applications, servers and network devices. They can tell you about your IT infrastructure and the behavior of your users and would be attackers. Log data is vital to maintain uptime, ensure system security and meet compliance mandates. Every environment has its unique footprint of IT data that can be leveraged. Examples of critical activities that use logs: Monitoring for issues before customers services are impacted Troubleshooting IT problems Conducting security forensics investigations Reporting on compliance mandates Logs also contain useful data for higher level analytics and reporting: Track patterns of usage for a particular service or user Provide service assurance and revenue assurance Trace complex patterns to enable detection of more sophisticated events Enable SLA tracking and reporting From operational management to customer usage stats, as IT infrastructure become more dynamic, getting visibility and making better use of your logs becomes indispensable.

16 NEW APPROACH: no Data Scientist needed!!! no transformation in ETL no data marts, star schemas, pre defined queries, etc. information forwarded, indexed storage managed between online use cases, vs. low cost, infrequently used data data enriched on the fly change the definitions and enrich the data in real time nothing but assuredly useless data should be thrown away during the process We must be able to create new dashboards in minutes. Again going back to how long does it take to use big data, I can create a new dashboard to share in 5 minutes that is from when I get the requirements to shared solution. My dashboards can be real time. I can create a dashboard that operates not on past data but on my bid data as it goes by. Charts, tables, graphs, Google maps, visualizations, all updating in real time.

17 And the ability to quick share them with appropriate access controls. 14

18 Splunk enables you to integrate external data sources at search time. For example, correlate external business data to IT data, run computations and discover new intelligence. This example is based on a low cost long distance telecoms company, where multiple carrier partners exist to route calls to a destination. In this real world example, Splunk performs an external lookup of partner tariffs from an external RDBMS and calculates the partner carrier fee for using their network based on actual call detail records. Performing this at search times helps mitigate risk in a variable pricing environment, where tariffs may change frequently. By doing this, Splunk is able to pinpoint the lowest cost partners for a specific customers destination, enabling the telecoms company to optimize their partnerships and consequently revenue. This is one of many use cases of Splunk correlating business and IT data to help optimize the business. Another is providing dashboard visibility into Average Revenue Per User (ARPU), an important KPI for telecoms businesses. The Splunk solution in this particular example took 1 week, whereas traditional approaches to this problem were taking many weeks without success with Splunk, there is no need to write database queries, or buy expensive reporting packages to query/report or visualize your data. At a time when the business is demanding more from IT at lower cost, Splunk provides a fast, flexible way to unlock powerful new intelligence from your 15

19 business and IT data. 15

20 Machine generated data while challenging to manage can yield insight to drive your business and uncover new opportunities From searching thru data to listening to events in real time From correlating logs to managing business outcomes From tracking customer behavior to improving customer experience 16

21 2700+ licensed customers in 74 countries $66M in revenues for 2010; 96% growth 900+ new customers in ,600 customers in 78 countries 50 of the Fortune

22 18