Data Challenges in Telecommunications Networks and a Big Data Solution Abstract The telecom networks generate multitudes and large sets of data related to networks, applications, users, network operations and call processing. This large data set has the capability to give valuable business insights - for example, real-time user quality of service (QoS), network issues, customer satisfaction index, customer churn, network capacity forecast and many more revenue impacting insights. The traditional telecom application architectures and solutions utilize central network management & monitoring servers, RDBM databases in operator s Network Operations Center (NOC). These systems are not designed for high scale and processing large sets of data for deeper insights and analytics, due to which such data intelligence is either lost or not fully utilized. Insight from this data can be utilized to improve user QoS, ensure conformance to SLAs, launch new services and products based on subscriber preferences, and their location and social preferences and based on their associations to other subscribers. This white paper will present Incedo s solution architecture using modern Big Data systems and frameworks for mining, reporting and presenting insights into ever increasing very large sets of data using open source and commercial tools that employ massively parallel processing engines and distributed databases, with focus to retain operator s Capex and Opex lower while providing deeper insights into their own data.
Introduction In this paper, we present various sources of data that s present within a telecom network data that s generated from end devices, network elements and management systems and describe the traditional data analytics solutions. We will show how and why these traditional systems lack in providing deeper insights into the data, due to both the inefficient data storage and processing architectures and lack of rich sets of analytical tools mainly due to inefficient data processing systems. We also present how these architecture and system level issues are addressed by modern Big Data architectures and frameworks. We then present Incedo s Big Data solution for telecom data, with information of different frameworks and tools that are available currently and the various choices we made in our solution. We present Incedo s solution that addresses both the high demands of data storage and analytics of very large scale data using a mix of open source and commercial based massively parallel processing frameworks and distributed data storage systems, to provide rich and realtime insights into Operator s own network data with lower Capex and Opex. Data in Telecom Networks and its Uses network failures, real-time measurements of subscriber call QoS, and monitoring and ensuring SLA conformance. The data from call data records in wireless networks, and session data records in VoIP networks can be analyzed for insights into real- time fraud detection, for example based on signaling message redirections, sim swap detection. The same data, along with external data sources like geographical information, social networks can be analyzed to prepare tailored marketing campaigns targeted to specific subscribers, in real-time, based on their location, preferences, and based on their social network associations. In addition, the data analysis can give insights for the operator and help in innovation of new products based on the stories the data is telling about users, applications, preferences. Telecommunications networks, both wireline and wireless networks, ranging from TCP/IP networks, VoIP networks overlaid over TCP/IP networks to 3G, 4G wireless access networks generate lot of data data from end devices to network elements. This data has a lot of information embedded into it - data related to network operations, individual users' application preferences, and interests to data that can be inferred about customer satisfaction with the network and operator's service to real-time market forecasts. Operators have this rich set of data which is very powerful and the data is the new gold that can give information and insights into their networks, subscribers, business, and insights into future products that will make the operator ever more successful. The data from telecom networks can be analyzed for network operational issues and o p t i m i z a t i o n s r a n g i n g f r o m r o u t e optimization, automatic re-routing on
The data comes from many sources like logs, SDRs/CDRs, events/alarms from network elements; signaling messages and application data streams from subscriber end devices; transactions, billing records, call durations, call pattern data from operator's own OSS/BSS systems; subscriber social preferences, associations data from social media, emails, SMS messages. Although the data has rich information and can give deep insights for the operators, the data can run into terabytes to petabytes of data per day based on the network size and the based on different types of data that's gathered. To process such extremely large data, not only the right set of data gathering, processing and analysis tools are required, but also highly scalable and massively parallel processing systems are required. Figure 1: below shows an example of traditional data analytics tools and systems in a typical operator's network operations environment. Figure 1: Traditional data analytics system in telecom networks In the next section, we describe why these traditional data systems as shown in the figure below, cannot do such complex data analytics, and in later sections, we describe how the new big data architectures are the right tools and systems for this need.
Data Challenges in Traditional Telecom World As the number of end devices (desktops, laptops, notepads and smartphones) and rich media applications are ever increasing, the wired and wireless networks and network nodes not only have enough challenge in handling the signaling and data from these devices to support the standard services, the amount data that needs to be stored & analyzed by network operators is growing exponentially and the complexity of such analysis is also becoming extremely challenging. Figure 2 below shows some of the challenges in analysis large sets of complex data in traditional data analytics systems. Telecom network elements (for example, enodeb or MME or SGW or PGW in a 4G LTE network) in a typical deployment of few hundred to many hundred cells and few thousands of subscribers will generate log data for recording the ongoing signaling and data activities, to help with troubleshooting the system in case of service or software issues. And these telecom network elements, as a total, can generate many 100s of MBs to GBs of data per hour. Many of the times, all this data will be streamed to servers in Network operations center for live and post processing. Another example is, where these network elements generate logs related to call events (for example, SIP messages related to a call flow) and all intermediate events that occur during the call, including account start/stop events which are written to CDR or SDR records or files. With a telecom system as whole, that can serve millions of calls per hour, the amount of call record data can run into many terabytes per day. Figure 2: Challenges in traditional data analytics systems
Retention of such large data, ability to efficiently search and mine such extremely huge data, and generate real-time business reports on such data has been a huge challenge (and in many cases still a challenge in many telecom operator environments), as the data can't be retained beyond few hours to few days, searches and reports take linearly long time, few hours to many hours all discoursing as it impedes real-time operations and business analysis and operators will be unable to take timely recovery actions. beyond which the performance can't be improved. In addition, from software release to release, the diagnostic log format or call record formats keep changing and this information of record format changes are propagated from vendor to operator to adjust their tools, in many cases requiring new software releases in both the places and causing the engineering and deployment overheads associated with new releases. Traditionally, these extreme volumes of data is Also, the tools that parse these logs and call analyzed in a central data center in operator's records are either custom developed by NOC, using stacked server blades and RDBMSs operators or by 3rd parties outsourced to by the for metadata and table data storage and SAN or operator increasing Opex, time to market and RAID arrays for raw data storage. These unavoidable dependencies all causing systems can only provide linear performance increased costs and business risks all till a point, where the CPU and network amounting to Big Data challenges. latencies of SAN become the bottlenecks, How Telecom Traditional Tools Won't Work Traditional data analytics systems in a typical telecom operator deployment, use RDBMs for storing data related to configuration, network discovery, faults and alarms, network diagnostic logs, CDR logs, CDRs, operational metrics, etc. all in the RDBMS in tables. And all this data is destined to operators' EMS/NMS systems. First issue is significant part of the data is nonnetwork management data. For example, although configuration, faults and alarms are data related to network management, data related to operational metrics, CDRs are operations data and give insights into historical operations and helps with troubleshooting networks. The diagnostic logs and CDR logs, media data give real-time streaming information about call quality, real- time network and system issues. In typical telecom network deployments, above 90% of data is non-network management related, but it's being stored at the EMS/NMS systems. As different types of mixed data sets are destined to same system and same RDBMs, with each data set being stored in a separate table, the non-network management tables continuously grow to millions and billions of rows causing the Big Table issue. As the RDBM's table size increases, the storage and search becomes very suboptimal; reports and analytics tools take longer and longer to process the data, essentially incapacitating operators to build deep analytic tools and reports to mine the business and real-time insights.
Second issue is, due to sheer volume, data cannot be retained beyond few days on these systems as it starts to hit disk capacity of central NMS/EMS systems, essentially losing insights into historical patterns and long period analytics. Third issue is traditional data analytics systems in most cases are centralized servers and centralized databases in operator's Network Operation Center (NOC) and the system can't be easily scale up. The scale up is typically limited to a front end load balancer with multiple servers but still destined to a single cluster of disk raids and RDBMS instances causing bottlenecks at the data storage layer. Fourth issue is cost associated with RDBMs systems. As the data size and storage needs increase, more instances of RDBMs are required and the associated license costs and server costs increase Capex and Opex. How The New Big Data Systems Can Address the Telecom Data Challenges Effectively Big data challenges faced by telecom operators need the new age Big Data solutions. The first issue described in previous section, the Big Table issue is addressed by Hbase, where a big table with large number of rows is split into regions that are served by the region servers. Regions are vertically divided by column families into Stores, which are internally stored as files in HDFS. Each Region Server is hosted on a clusters, for example multi-node HBase and HDFS clusters in a Big Data architecture. With ever increasing data, the cluster size can be increased to retain large amounts of data for very long periods giving sufficient data for the operators for insights into historical patterns and long period analytics. physically different machine. The third issue of centralized servers and databases which become performance With the new data architectures of HDFS, bottlenecks is addressed by massively parallel Hadoop and Spark that enable massive processing systems that run on multi-node distributed storage, data splitting and parallel clusters. For example, Spark framework, with processing, the challenges of telecom big data cluster of nodes that orchestrate parallel processing are aptly addressed. The Hadoop processing and execution, provides job and task and Spark based systems will scale to provide schedulers that can execute queries in parallel consistent performance even when the on the database region servers and collect the number of data sources and the amount of results. The system also comes with a purpose generated data ever increase all due to built SQL query engine, SparkSQL, which distributed and parallel processing provides traditional SQL syntax but in the architecture. background executes the query in parallel on all The second issue of retention of data is addressed by employing multi-node database region servers in parallel, collects the results and responds to the query.
The fourth issue of high cost analytics systems is addressed by open source and cost free databases like HBase and HDFS, which also provide high scalability. Many of the new Big Data systems are open source, and further reduce the Capex and Opex costs for operators, while at the same time addressing the issues present in the traditional data a n a l y t i c s s y s t e m s. T h e s e B i g D a t a architectures also support redundancy, load balancing options further enhancing the system's stability and high availability. In addition, in Big Data systems like in the Spark architecture, the streaming data processing capabilities, iterative nature of map-reduce machine learning capabilities built into its framework, in-memory caching of data leads to many folds improvement in performance, and making it a reality for telecom operators to analyze their networks in near-real-time to real-time. For the rich user interface and dynamic reporting, many tools and platforms like JasperSoft, Pentaho are available for rich set of business intelligence analysis and reporting, which in the backend use SQL and SparkSQL queries to access data and run analytics algorithms to mine the data for intelligence and present in intuitive manner for operational and business insights. An Example Telecom Big Data Solution from Incedo The figure below shows an example solution from Incedo to address the telecom Big Data challenges; and also shows rich operational and business intelligent system for analyzing rich and deep insights into the data. Our solution uses multi-tier Big Data architecture as shown below. In this solution, we address a specific problem of telecom operators, where the issue is all of the data from the data sources is structured data, but the traditional data analytics systems had challenges in analyzing this data due to the Big Table issue and performance bottleneck issues due to central server(s) and database(s), both the issues that were described in previous sections. In this multi-tier Big Data solution, the data sources are the typical historical and streaming structured data in telecom networks like, CDRs, SDRs, logs, events, alarms, CDR logs and user data.
Figure 3: Incedo Big Data solution architecture for telecom network data For the Data Ingestion and ETL (extract, transform and load) tier, where the data is extracted, parsed transformed as needed and stored into appropriate data storage and databases for further analysis. For ETL function, there are many tools, both commercial and open source are available. For example, some of the commercial and proprietary ETL tools are Informatica, ODI and Datastage. Some of the ETL tools are open source but incur costs when productized, like the Pentaho ETL, Talend ETL, and then they are fully open source tools like Flume, Kafka. Or one can develop their own custom scripts in Python, Java and other scripting languages. In our solution, we have used custom Spark python scripts, as these scripts were used not only to parse, extract the structured data but also to generate aggregation reports on the fly as they parse the data and store into the databases i.e. one set of reports (aggregate reports) are prepared in-line with the extract and transform function. For Big Data System tier, where the data is stored and processed, there are many choices for the tools and frameworks. For the RDBMS based storage and processing, some of the available options are MariaDB, Inforbright, InfiniDB, Vertica, Amazon Redshift, Hana and Taradata. For the NoSQL based storage and processing some of the available choices are HBase, MongoDB and Cassandra. For simple file system systems, one can use HDFS. For the data processing engines, both open source and commercial tools are available. Apache Spark, for example, is an open source data processing engine while Amazon Elastic Search is a commercial option. Both are equally capable processing engines for Big Data analytics needs.
In our solution, we have used Spark clusters architecture components that provide with HBase database clusters on HDFS. For massively parallel processing capabilities with streaming data processing, we have used Spark dynamic scaling, redundancy, load balancing; Stream engine cluster. Main reason for this and one that provides dynamic and real-time selection is to keep costs low and attain high reports that give deep insights into the data processing performance as well as near- operational and business aspects of telecom real-time analysis of streaming data. networks ranging from user QoS, SLA conformance, KPIs, customer retention, real- The Data Access tier options depend on the time revenue forecasts. The solution can be data processing engine option that's selected. easily adapted to unstructured data or a mix of Most data processing engines provide SQL structured and unstructured data and can be syntax for queries, although systems like Spark customized to address other related Big Data provide SparkSQL library which provides Analytics challenges. The solution uses latest standard SQL syntax to an end user but components of open source big data systems to underlying uses Spark multi-node cluster and contain the Capex and Opex of operators, while massive parallel processing architecture of providing rich insights into their data and thus Spark to execute the SQL query on multiple helping Operator success. nodes in parallel, collect the results and report back to user. In our solution, we have used both SparkSQL for the spark subsystem that processes historical structured data, and SQL for the streaming structured data. For Reporting and Visualization tier, some of the available tools and frameworks options are JasperSoft, Pentaho, Kibana, Tableau, MicroStrategy, and Qlikview all of commercial tools. We found these tools are better in presenting and visualizing dynamic and sophisticated reports. Or one can develop a custom web application rather than using a third party Reporting and Visualization tool, although the cost and development time can become disadvantage. In our solution, we have used JasperSoft, as we found it provided the desired dynamic reporting capability with manageable costs. Incedo's Big Data solution for telecom networks provides an end to end solution using highly scalable new and latest data
Conclusions The telecom data challenges are real and data is growing larger and larger. With traditional data processing and analytics solutions, operators are unable to effectively use the data and information available to them and incapacitated to find deeper operational and business insights into their own data. Telecom operators need new solutions for the Big Data that their networks generate and the valuable information they have from their own OSS/BSS systems. Incedo solution provides a comprehensive solution that gives insights into operational and business intelligence; with latest and modern open source big data architecture involving massively parallel processing and distributed database frameworks and dynamic reporting and visualization. Thus our solution provides a comprehensive view and gives deep insights into telecom data with lower Capex and Opex costs. References: 1. http://www.bigdatatelecoms.com/ 2. http://www.strategyand.pwc.com/media/file/strategyand_benefiting-from-big-data_a- New-Approach-for-the-Telecom-Industry.pdf 3. http://bigdata-madesimple.com/11-interesting-big-data-case-studies-in-telecom/ 4. http://www.datawatch.com/how-telecom-companies-are-using-big-data-analytics/ 5. http://www-935.ibm.com/services/us/gbs/thoughtleadership/big-data-telecom/ 6. http://www.ibm.com/analytics/us/en/industry/telecommunications/ 7. http://www.telecomitalia.com/tit/en/bigdatachallenge.html 8. http://wwwen.zte.com.cn/endata/magazine/ztetechnologies/2012/no6/articles/201211/ t20121121_370620.html 9. http://wwwen.zte.com.cn/endata/magazine/ztetechnologies/2013/no6/ 10. http://wwwen.zte.com.cn/endata/magazine/ztetechnologies/2013/no6/articles/201311/ t20131115_412726.html 11. http://wwwen.zte.com.cn/endata/magazine/ztetechnologies/2013/no6/articles/201311/ t20131115_412729.html
NAGESH DEVISETTI Director - Wireless, Communication Engineering Email: nagesh.devisetti@incedoinc.com https://www.linkedin.com/in/nageshdevisetti MANISH GUPTA Vice President - Communication Engineering Email: manishg@incedoinc.com https://in.linkedin.com/in/manish-gupta-108a3319 About us Incedo Inc (formerly a part of $4Bn Indiabulls Group) is a technology solutions and servicing organization headquartered in the Bay Area, USA with workforce across North America, South Africa and India (Gurgaon, Bangalore). We specialize in Data & Analytics and Product Engineering Services, with deep expertise in Financial Services, Life Science and Communication Engineering. Our key focus is on Emerging Technologies and Innovation. Our end-to-end capabilities span across Application Services, Infrastructure and Operations. What really differentiates us is: Strong engineering talent Focus and passion for innovation Flat organization structure responsive engagement models Agile and flexible delivery and commercial models Focus on long term partnership with clients USA: 2350 Mission College Boulevard, Suite 246 Santa Clara, California - 95054 Tel: +1408 531 6040 INDIA: 248, Udyog Vihar Phase-IV, Gurgaon - 122 015 Tel: +91 124 4345900/01/02 www.incedoinc.com