How To Make Data Streaming A Real Time Intelligence

Transcription

1 REAL-TIME OPERATIONAL INTELLIGENCE Competitive advantage from unstructured, high-velocity log and machine Big Data

2 2 SQLstream: Our s-streaming products unlock the value of high-velocity unstructured log file, sensor and other machine data, insuring new levels of visibility and insight required to drive both manual and automated actions in real time. Unlocking the intelligence in machine data Businesses are moving from simple monitoring and searchbased tools, and trying to understand the meaning and causes of business and system problems. Operational intelligence enables organizations to make decisions faster and to act immediately based on real-time insights. However, making use of high velocity machine data to drive immediate and automated actions presents some real challenges. Existing data management and business intelligence systems, even including Hadoop, are not engineered for low latency operations from high volume and high velocity data. The definition of machine data covers, not surprisingly, all data generated by machines servers, applications, sensors, smartphones, websites, networks and services all generate vast volumes of data every second. It covers everything from data centers, telecommunications networks to machine-tomachine and the Internet of Things in a device-connected world. Log files are the most common source of machine data today across all industries. But sensor networks are catching up fast. And telecommunications has always been a generator of Big Data with Call Detail Records (CDRs), network equipment performance measurements and subscriber and handset location data. Machine data contains a wealth of information, for example, on consumer behavior and location, customer quality of experience, financial transactions, security and compliance breaches, as well as the state of industrial processes, transportation networks and vehicle health. Hidden in the data is critical intelligence on the real-time performance of business processes and operations. For example, machine data contains the keys to user experience and behavior, service level breaches, fraud and security breaches, transportation network flow, network and service performance. That s why machine-generated data is the fastest growing and most valuable area of Big Data. Operational intelligence is the discipline of collecting and extracting intelligence in real-time. To be useful, intelligence must be delivered to business people in real-time and be easy to use. Our s-streaming products offer a better, faster and lower cost way to harness the intelligence stored in machine data flows, and, with continuous integration of streaming intelligence across the enterprise, SQLstream is transforming how businesses operate in a real-time world.

3 3 SQLstream is the leading real-time operational intelligence platform. It is built on a massively scalable, distributed platform for processing live streams of machine data in real time. Any type of machine data can be collected; sophisticated predictive analytics and advanced pattern detection can be applied across any and all streams simultaneously, with on-the-fly visualization and the ability to integrate streaming intelligence with Hadoop, other Big Data storage platforms, enterprise systems and data warehouses. Security intelligence SQLstream s operational intelligence platform transforms log and other machine data into realtime security intelligence. Where traditional SIEM and log monitoring tools fail, SQLstream manages with ease the increase in data volume and velocity coupled with more sophisticated known and unknown threat detection scenarios. Infrastructure and operations Real-time monitoring with predictive alerts for outages and SLA breaches, utilization and security breaches, collecting log data in real-time across all IT operational siloes, including cloud and virtualized environments. Financial services and low-latency trading Continuous aggregation with ultra low-latency analytics is a fundamental requirement for many financial services applications. Analytics must be derived from multiple sources of diverse streaming data including trading data, news feeds, credit ratings, transaction data, customer data, market data and fraud patterns in real-time. Customer QoS from CDR Data Telecommunications generate a vast amount of network, service and customer data. In particular, each call or data service generates multiple call data records (CDR), which when collated and analyzed in real-time, offer immediate insight into network and service quality, and user quality of experience. Monetization of M2M data streams Machine-to-Machine mobile services such as logistics, healthcare, smarter home and smart energy applications generate a vast volume of real-time data. SQLstream provides a massively scalable platform to both collect all sensor data in real-time, and transform it into streaming intelligence on users, quality of service and location. GPS and location-based analytics Vehicle and smartphone GPS data offers real-time insights into user and traveller behavior and location. GPS data can be transformed into real-time traffic flow and congestion maps, and used to detect user location for improved delivery of location-based services. SQLstream: real-time scalability and automation for a wide range of industries and applications.

4 4 Our customers have eliminated the chasm between analytics and their operations, using SQLstream to connect their Big Data storage platforms directly to their data. SQLstream generates real-time operation intelligence from machine data, and integrates data and streaming intelligence seamlessly with existing storage and operational platforms. So what makes SQLstream different?! Instant results. Unlike other products that must store data first, SQLstream s streaming data management platform ensures ultra-low latency with real-time immediate insight.! Instant productivity, no hidden obstacles. SQLstream is built on standards-based SQL. Applications can be deployed and extended easily without low level coding or proprietary languages.! Rich data analytics. SQL is a powerful and expressive language for data management analysis. Unlike other tools with weak and simplistic languages, SQLstream enables powerful time-based and geospatial queries over all data sources simultaneously.! Low cost with proven Big Data scalability. SQLstream offers low cost yet Big Data scalable solutions for extracting operational intelligence from high volume, high velocity data.! Complements existing systems. SQLstream enhances existing systems and architectures, allowing current investments to remain in place while deploying SQLstream as the real-time operational intelligence layer on top.! Join and correlate across all data. SQLstream collects and joins data across all formats and types, enabling any complex exceptions and patterns to be identified.! Continuous integration of streaming intelligence. There is a strong correlation between organizations that are utilizing operational intelligence successfully and those who integrate their operational intelligence capability with existing enterprise storage platforms, middleware and other infrastructure systems. SQLstream solves the advanced problems that are out of reach for other log monitoring and search-based operational intelligence tools. SQLstream s streaming data management platform offers scalability for high velocity as well as high volume data, plus streaming SQL for powerful data manipulation and analysis of data streams. Enterprise requirements Operational Intelligence with SQLstream In summary, SQLstream can turn machine-generated Big Data into real-time value by generating real-time operation intelligence from live machine data, and integrating data and streaming intelligence seamlessly with existing platforms.

5 5 Streaming SQL and Real-time Hadoop The need for low latency, real-time operations is emerging as a primary new requirement for Big Data, and streaming is the key capability required to support these real-time business processes. In parallel, as Big Data technologies mature through wider enterprise adoption, SQL is emerging as the de facto language for enterprise Big Data. SQL provides simpler, high performance and reliable queries. Hadoop is engineered for processing high volume, unstructured data quickly once the data are stored, but has high throughout latency making it unsuitable for real-time, low latency use cases. SQLstream s core s-server platform uses standard SQL queries to process live data streams. SQL is the ideal language for processing data streams using real-time, windows-based queries. The issue Streaming SQL Views on Hadoop with normalization and rigid schemas is a non-issue for a streaming data platform there are no tables, and no data gets stored in the SQLstream streaming layer. COLLECT ANALYSE SQLstream RT Dashboard SQLstream SHARE Impala / HIVE HBase HDFS / Map Reduce Hadoop and SQLstream have similar characteristics and architectures for scalable, distributed processing of machine data. However, where Hadoop excels at batch-based processing, SQLstream excels at streaming data processing and operational intelligence. SQLstream s streaming Connector for Hadoop HBase eliminates the 5 chasm between operational intelligence and Big Data storage. Data and operational intelligence can be streamed directly into Hadoop HBase for storage and further analysis, equally data can be streamed out of HBase, either to be joined with real-time data, or replayed through SQLstream s operational intelligence platform for further analysis. Copyright [email protected] Organizations are able to exploit the value of both real-time analytics over the arriving data, and combine the arriving data in real-time with processed, filtered and aggregated trend data from Hadoop HBase. Operational intelligence results are enhanced by combining real-time data against known trends, eliminating false alarms and longer term comparisons. The extraction and data processing in SQLstream uses standards-based SQL queries, enabling powerful real-time queries to be deployed over streaming HBase data.

6 6 ROI SQLstream s customers generate ROI through faster detection of business and system issues, the ability to deploy low TCO and faster solutions, and to improve overall operational efficiency through continuous integration of streaming intelligence. Security incidents, for example, can be identified in seconds rather than hours or days, quality of service breaches predicted and corrected before the incident can occur, consumer behavior analyzed in real-time, and customers presented with appropriate real-time promotions.! A leading telecommunications company was able to develop complex time-based pattern analysis for predicting 4G network call failures in real-time. The resulting solution was developed using a fraction of the code and with an order of magnitude performance advantage over alternative products.! One of the leading US HPC centers was unable to find a log monitoring tool that could scale to their high volume, high velocity requirements. SQLstream enables them to monitor and analyze all their log data in real-time.! One of the largest cloud infrastructure providers was able to predict runaway application processes in real-time, preventing overall platform performance degradation and impacting customer satisfaction through excessively high billing.! One of the world s leading government transportation agencies saved over $10M by implementing a real-time traffic congestion detection system based on SQLstream and GPS data, rather than the traditional approach of installing road-side and in-road sensors.

7 7 FREE DOWNLOAD SQLstream s-server, our core streaming Big Data engine is available to download and trial for free. You get the fully featured SQLstream s-server product along with access to the full documentation set and example applications. You will be able to use in the product for 60 days without any restriction. UNIVERSITY PARTNER PROGRAM SQLstream is dedicated to developing cutting-edge, open technology while supporting the advancement of research through collaborative innovations throughout the Big Data ecosystem. For recognized universities and colleges who are looking to expand research into streaming Big Data concepts, the SQLstream University Partner Program offers a non-commercial, royalty-free license to our technology. As a SQLstream University Partner, undergraduate and graduate scholars have access to the full depth and breadth of real-world streaming data knowledge as shared among other educational institutions and participating enterprise organizations.

8 SQLstream, Inc Market Street San Francisco, CA,