Augmented Search for IT Data Analytics New frontier in big log data analysis and application intelligence Business white paper May 2015
IT data is a general name to log data, IT metrics, application data, monitors and other machine-generated data. IT infrastructure either private or hosted on public clouds generates huge volumes of unstructured IT data. IT Data is big data due to its volumes, unstructured and dynamic nature and distribution across multiple physical and virtual sources. Critical Data and Intelligence Every tier in the IT infrastructure generated logs, metrics, databases records, and other events. Analyzing this data can help users understand: 1. Statistics trends, application intelligence, and IT to business correlation 2. Web Analytics web logs with rich analytics information 3. Application quality and performance quickly find errors, problems, and anomalies. 4. Isolate problems faster, prevent loss of business 5. Investigate security and fraud quickly 6. Track and investigate business transactions problems. Analyzing IT Data Complex IT architecture combined from many devices, servers, software, virtualization and application. IT administrators investigate through huge volumes of log data every day for troubleshooting, security and intelligence needs. The challenge is have a single platform that can handle all data types and sources, correlate them and provide out of the box value for any type of IT component. Augmented Search for IT data XpoLog Augmented Search brings a new technology and approach to the big data management domain. Augmented Search combines end-to-end log analysis and management platform with cutting edge machine learning and automated Analytics. The new solution delivers: Log management and analysis of data across all sources Super-fast search for manual investigation and troubleshooting 2
Automatic visualization of complex log data for reports and dashboards Proactive monitoring Automatic Analytics engine, executing many algorithms for machine learning, data mining, statistical analysis, semantic analysis, discovery, and profiling The result is a built-in intelligence engine that supports user decisions during analysis, proactively adding layers of information and helping to understand data faster and in an automated fashion. This approach is technology oriented and does not require tags, filters, or predefined rules. It supports both homegrown solutions and 3 rd party applications. Augmented Search helps organization generate ongoing value from log and machine data without manual work. Augmented Search combines a super-fast search with super-smart Analytics. IT Data Analysis Platform The XpoLog log data analysis platform is built over the following primary components: 1. User interface Web user interface with built-in Tomcat server 2. Dashboards and reports 3. Virtual Data Engine log access, collection, parsing, and management 4. Indexing and Search engine 5. Analytics engines many algorithms that auto generate intelligence This high-level architecture has subcomponents that handle security, proactivity, system health, self-healing system, map/reduce management, connectors, and more. Please visit http://wiki.xpolog.com for more information. Augmented Search for IT Once XpoLog Augmented Search is deployed all data will be collected and analyzed. The following participants will experience great value from using XpoLog: 1. System administrators 2. Web servers and App server infrastructure team 3. Support engineers 4. Developers and Testers 5. Business application owners 6. Production support 7. DevOps 3
8. Operations 4
Application data will be securely visible to the right people for search and analysis. Auto detected analytics will present problems, trends and transactions to users. XpoLog will generate live dashboards and reports for ongoing intelligence. Managers can take better decision with improved understanding of the application state. The XpoLog platform can digest both structured and unstructured data from multiple sources. By indexing and analyzing log and business data, it is possible to create a rich set of intelligence reports, both on common data format and homegrown generated data. Some examples from our customers: 1. Servers usage analysis, windows events collection. 2. Web analytics, trends, and web app usages 3. Trade transaction statistics, volumes, and trends 4. Application features profiling and statistics 5. Ecommerce web site business intelligence ROI Augmented Search helps troubleshoot faster, and automatically visualizes complex data. With constant proactive Analytics, you can support homegrown solutions and 3 rd party applications, making TCO much lower and saving exhausting manual work. This combination of automated intelligence with super-fast search and dashboards drives efficient out of the box value quickly. Customer Use Cases Troubleshoot Virtualized Infrastructure Quickly search event types, errors or abnormal pattern, transaction Ids, users, errors, exceptions, etc. across all log data and sources. Search for MAC address with IP address and a corresponding user ID and focus your search query on specific server or log. Instead of manually connecting many servers and logs, search all the data in one place. 5
Proactive problems detection - Augmented Search Augmented search helps to quickly find problems in the application tiers. Once user search or navigate the application data (web, apps, os logs and so on) augmented search will present a summary of problems and anomalies that were detected in that app log data. The augmented search intelligence layers help to focus on the important things first, without the need to read millions of log events. Performance and Analysis Reports With XpoLog complex analysis search, users can visualize data automatically. You can measure response time between different log events, create summaries of thread activity, memory allocations, exceptions and more. Understand the trends and bottleneck of you application tiers across the board. Investigate and Analyze Transactions Correlate log events and build advanced visual transaction search engine. User our transaction analysis options to track and correlate different events across logs and servers. XpoLog can measure time and data integrity in the transactions, save hours of work by defining flows and custom dashboards with statistics and transaction metrics. Trend Analysis on Windows Events By collecting and analyzing all windows log events, simply create statistics dashboards on user activity, application problems and log generation ratio. Summary Augmented Search is a unique technology based on our deep understanding of IT data and organization use cases. The technology helps organizations build a more robust ROI oriented strategy towards big log data in today's data centers. We invite you to read more about technical features in our data sheets and documentation. 6