Graph Analytics: Beyond the small circle of friends

Transcription

1 Graph Analytics: Beyond the small circle of friends An Ovum white paper for YarcData SUMMARY Catalyst Ovum view Big Data has raised the bar for analytics: organizations need no longer make the tradeoff between depth of analysis and breadth of data ( rich vs. reach ). New platforms and frameworks are making possible the ability to solve new kinds of problems, such as analyzing real world scenarios involving networks of people, places, or things connected by many-to-many relationships that are impacted by events, economics, natural phenomena, or other trends. Graph analytics have emerged to help organizations make sense of these trends, yielding 360-degree views of all the relationships involving an entity and the ability to explore complex patterns in those relationships. Yet early approaches to graph computing have largely been restricted to the intelligence community or social network analysis because their compute-intensive nature has limited them to piecemeal, batch processes. If performance and capacity limitations to graph solutions could be overcome the benefit of graph analytics could be applied to w wider variety of scenarios as capital markets risk management, life sciences research, healthcare delivery, smart infrastructure management, and other areas. Fast Data the velocity component of Big Data is important to unlocking the value of graph analytics. Because most real world scenarios are dynamic, graph computing solutions must be sufficiently powerful to handle the entire problem without breaking it into multiple parts, and must be able to compute the entire problem without partitioning it, and deliver results faster than batch mode because of the changing nature of the scenarios being studied. Speed is also critical for handling the continuous flow of new data, because the world, and all the relationships that exist, are constantly in changing. That in turn makes it essential to perform rapid querying so that the results do not fall out of date. New, optimized system or appliance oriented approaches could help break the graph computing bottleneck, enabling these solutions to address a far wider range of business, infrastructure, and political scenarios. Key messages Big Data and Fast Data are changing expectations around analytics Page 1

2 New analytic approaches, such as graph analysis, are emerging to address new kinds of problems not otherwise possible with conventional data warehousing platforms Graph analytics is a new approach for analyzing the complex networks of many-tomany relationships that pervade the real world Graph analytics is a prime example of the type of analytic problem that benefits from a Fast Data approach New, optimized system or appliance oriented approaches could help break the graph computing bottleneck BIG DATA AND FAST DATA RAISES EXPECTATIONS A new analytics norm is emerging Big Data has raised the bar for analytics: organizations need no longer make the tradeoff between rich and reach. They no longer need to limit reach (sample size) when running rich, sophisticated analytics, and they can now extend analytics to other forms of data beyond the structured data that have populated enterprise data warehouses. Big Data defined by Ovum as embodying the 4 Vs including volume, variety, velocity, and value is now fair game for analytics. By expanding the range of addressable data, Big Data has raised expectations for delivering superior outcomes for analytics aimed at use cases such as improving customer retention, managing public infrastructure, protecting the nation s security, and optimizing healthcare delivery. Big Data has also raised the appetite for Fast Data, which addresses data velocity. While there have always been specialized applications for consuming real-time data feeds, growing bandwidth, memory density, and continuing trends with Moore s Law are in turn feeding expectations for realtime analytic solutions that not only improve performance, but allow greater adaptability and flexibility. New data, new analytic approaches While Big Data has improved the fidelity of analytics with many familiar problems, new platforms and frameworks are clearing the way for solving new sets of problems that were traditionally beyond the capabilities of the largest and fastest conventional data warehousing platforms. There is the need for organizations, not simply to explore data, but gain a better understanding of the environment in which they operate. Typically, the real world is characterized by many-to-many relationships; examples include: People A person has relationships, not just with a single person, but with different groups and things. For instance, a person who buys fertilizer might not be of interest unless he shares a post office box with another person who owns a truck and was recently detected by video cameras loitering around Times Square. Products -- The success of a product depends, not only on the perception of people in its target market, but the introduction of other products, the ability to make delivery to Page 2

3 specific regions, changes in consumer buying power, and news events that cannot always be anticipated. Trading Risk When complex instruments are traded in capital markets, buyers and sellers must manage counterparty risk. In today s fast-paced financial markets requires an approach that assumes that relationships between parties and counterparties are continually changing. Modeling and analyzing the complex, many-to-many relationships that are present in the everyday world proved an extremely difficult problem for existing relational database platforms. At minimum, computing such problems would have required multiple join operations exacting huge overhead on performance. In practice, analytics involving complex webs of relationships was broken up into multiple discrete runs, each of them number crunching portions of the problem, yielding vague, approximate conclusions at best. INTRODUCING GRAPH ANALYTICS The wave of innovation surrounding Big Data provided early answers to this puzzle. A number of open source graph databases have emerged, that are often run to analyze social tribes that populate social networks; arguably, many of them cannot handle the volumes or velocities of Big Data. In effect, these analytics apply the technique of analyzing focus groups to the real world: identify key people in a demographic and identify the groups where they belong and the thought leaders who influence them. Unlike relational databases that represent data in tables that resemble spreadsheets, graph databases depict entities as nodes with unique properties and pointers to related elements that in aggregate form interconnected networks. As such, an entity or node has its own attributes and can have relationships with multiple other entities, and vice versa. A simplified representation of a graph model for a social group or tribe is shown in Figure 1. Yet, the well-known example of mapping social tribes with graph analytics represents only the tip of the iceberg of what s possible. Until now, the challenge with graph analytics that has largely limited it to rudimentary uses such as social networking is that they are extremely computeintensive, or extended use cases such as mapping homeland security threats. With the softwareonly solutions that have been available to date, these computations have had to be processed in batch mode. Furthermore, capacity issues forced the need to partition graph processing to different nodes in a cluster. Page 3

4 Figure 1. Basic graph model for social group YarcData s systems approach to graph Big Data analytics emerged initially with scaled-out clusters containing commodity hardware. Such approaches have been inefficient for graph analytics because they were optimized for sequential file access computing patterns. Graph computing requires, not only the ability to massively parallel processing, but also the ability to pool large, un-partitioned problems over large multiprocessor architectures and shared memory that is linked through high speed interconnects. YarcData adopted a systems-oriented approach to address graph computing bottlenecks. Its solution, urika, includes specialized hardware and software. The hardware divides the workloads between specialized accelerator nodes that are optimized for graph processing, and service nodes that handle core systems housekeeping chores such as appliance management, database management, and visualization. The system, which can accommodate up to 8,192 graph accelerator processors, can run jobs un-partitioned thanks to access to up to 512 TB of shared memory. Because processors and memory have different throughput rates, the system has been optimized to overcome latency through massive multithreaded techniques that expand effective processing bandwidth. On the back end, urika packages a parallel file system that can scale to exabytes. urika is tuned to run a software stack built on industry standards that is compatible with the front end of the open source Apache Jena graph database. Jena was designed to use RDF (Resource Page 4

5 Description Framework), a W3C metadata modeling standard for exchange of data on the web; and SPARQL, a protocol for remotely triggering RDF queries. The back end was optimized for YarcData s accelerator processors. Figure 2. Map of interactions between proteins Source: Institute for Systems Biology, courtesy of YarcData urika is the first appliance specially designed for applying a Fast Data approach to graph processing. As a result, urika opens graph processing to a wider range of potential use cases. Example: Improving healthcare delivery A federal science and technology contractor has been exploring graph analytics for addressing a wide range of problems, such as improving telecommunications network resiliency or accelerating life science discoveries. Recently, the organization has explored the potential of implementing the urika appliance to help optimize the prescription of drugs for evidence-based healthcare delivery. The challenge is that the information must be pieced together from multiple online sources including electronic patient records that list the disease and treatments; Medicare or other databases that track care delivery; databases that describe drugs; and in some cases, separate databases that track clinical trials. The scenario is a classic linked network with interdependencies between each of the nodes. Variables could include patient age and gender, the outcomes with specific drugs (or combinations of medications), specific dosages, stage of disease when administered, and potential drug interactions. Computing such a problem would be practically Page 5

6 impossible with conventional SQL databases because of the need for deeply nested joins, while traditional software-only graph implementations could not deliver the required performance. The organization is looking to urika to connect the dots interactively, so practitioners can make smart, evidence-based prescriptions on the spot. RECOMMENDATIONS FOR ENTERPRISES Author The public emergence of Big Data has raised the bar of expectations for what is possible with analytics. That encompasses the ability to make smart decisions in the real world, where many-tomany relationships are the norm. The surface is only just being scratched regarding the potential of graph analytics to help organizations make smart decisions in a complex world where dynamic, many-to-many relationships are the rule. Most of the analytics to date have been relatively static, requiring extended batch processing runs. Fast Data approaches are critical to expanding the range of graph processing beyond niche tool for social tribal analytics. The possibilities are especially broad as Fast Data approaches to graph processing are emerging to make this form of analytics responsive to a changing world. There will be a learning curve as the technologies and practices are new. But for organizations that are seeking a competitive edge in addressing so-called intractable problems of the real world, now is the time to start learning. Tony Baer, Principal Analyst, Ovum Enterprise Solutions tony.baer@ovum.com Ovum Consulting Disclaimer We hope that this analysis will help you make informed and imaginative business decisions. If you have further requirements, Ovum s consulting team may be able to help you. For more information about Ovum s consulting capabilities, please contact us directly at consulting@ovum.com. All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of the publisher, Ovum (an Informa business). The facts of this report are believed to be correct at the time of publication but cannot be guaranteed. Please note that the findings, conclusions, and recommendations that Ovum delivers will be based on information gathered in good faith from both primary and secondary sources, whose accuracy we are not always in a position to guarantee. As such Ovum can accept no liability whatever for actions taken based on any information that may subsequently prove to be incorrect. Page 6