Client Overview Our client is one of the leading providers of business intelligence systems for customers especially in BFSI space that needs intensive data analysis of huge amounts of data for their decision making. Our client s products aims at overcoming the limitations of regular business intelligence solutions like integrating data from different data sources, reducing data redundancy, etc. Our client s products utilize the benefits of modern technologies and implements graph traversal techniques that extract valuable information in days instead of weeks. Engagement Situation In today s rapidly changing environment, data analytics has become imperative to extract meaningful information from the raw data available in different form to meet the business requirements of any organization. Our client s customers wanted to have a capability of processing data extracted by using NoSQL databases and graph logical structures. Client s existing product had issues with scalability to keep pace with the ever increasing amount of data. Also, it was difficult to clean, convert and optimize the incoming data due to different sources and formats in which data was required to be extracted. Key Requirements Develop a scalable and optimal solution BI solution with: Data Extraction: retrieval of data from different data sources viz. DBMS, CSV, and text files in a specific pattern. Cleaning and fetching data with relatively few computational operations, and produce actionable output data. Develop reporting and provide real-time analytics and visualization on the processed data.
Xoriant Solution Xoriant has significant experience in database management systems. For the current engagement, the team studied the existing data processor to understand if the current architecture could support the new functionalities and if it could be made scalable. In reference to the requirement of the system, Xoriant team defined an optimal solution that accesses data across the multiple layers using Hadoop as the technology for data processing and storage. HTML 5 was used for visualization module for charts and tables displayed as different graphs e.g. line graphs, bar charts, pie charts, etc. A central graph logic based controller was developed which coordinates and makes the required changes across the data stack based on predefined rules. Xoriant Key Contributions Xoriant implemented a scalable solution which provided the required functionalities of eliminating large amount of code at each layer of present system by using Cassandra DB. Integrated client s platform with Hadoop framework while using Cassandra, a NoSQL database and Solr text search platform for processing data effectively. Incorporated NoSQL database, graph logical structures and graph traversal technique to enable validation, cleanup, and manipulation of data before it can be used by client s system. Integrated CassandraBulkOutputFormat and SolrBulkOutputFormat packages to write data in Cassandra and Solr. Integrated D3JS and Node JS library for visualization in the form of charts like line graphs, bar charts, pie charts, etc. with real time data. Used centralized graph algorithm to develop customer data connector and a loading framework to load data into the Graph storage.
High-Level Architecture Diagram Data Source (CSV) Tools and Technologies Java Cassandra Solr Hadoop HTML CSS Javascript NodeJS
Business Benefits Provided high scalability as any new data node can be added and related to current node in the system. Reduced the implementation and ongoing maintenance cost by 20% because of the centralized graph logic algorithm. Introduced new reports and reduced report generation time by 70% resulting in quick informed decisions. Resulted in highly optimized and efficient system in terms of time and storage space with customized graph searching functionality. Enabled the system to build, configure and solve complex analytical problems for e.g. Market Basket Analysis, Cluster Analysis etc.