XDR. Big Data solution.

Transcription

1 XDR Big Data solution.

2 MAIN GOAL Xdr is a solution that uses a simple low-cost architecture integrated to the business infrastructure of Telecom companies. Both Telcos and Internet businesses have often been requested to issue XDR reports (Communications and Internet Services reports) which involve long periods of time (they may be months or even years) which demand the ready availability of historic data. These requests are made by government organs (the Judiciary, the Police, etc.) and are supported by legal regulations. In view of this, the companies must be prepared to provide: Details and information of the customers traffic (VOICE, MMS, GPRS). In some cases, to provide the retention of up to 10 years of historic data. To provide information about IP-based services. All of these requests must be made with high security levels and must also provide auditing capability. The cost of adapting the systems to retain the customers historic data is substantial; while the implementation of generic solutions reaches the desired goals and requirements almost immediately thus minimizing potential penalties. Below we have the logical sequence of XDRs processing: XDRs are converted into a single layout which contains the requested information including several types of files (SMS/MMS, GPRS, FIXED VOICE, MOBILE VOICE, IP- SERVICES). Next, the files are transferred to a Windows Server. A batch program run on the Windows server and begins the data compression using algorithms with a high compression rate, thus reducing by 87% the size of the original files. In a batch execution mode, these files continue being handled with actions such as the removal of duplicated XDRs and grouping by customer thus reducing even more the size of the files. The compacted files are in a compacted format which makes it impossible to access them without using the xdrsearch module. 02

3 main features What would be more important for a produc that will search tera or even petabytes of historic data? We can affirm that it is the data compression rate and, next, the speed to search these data (breach of secrecy). Xdr has the following characteristics as far as these requisites are concerned: Compressing may reach 87% of the original file size, that is, a 100 byte file is stored within 13 bytes. That is a relevant reduction as far as hard disc consumption is concerned which means significant savings. The search speed of one month of Xdrs for any given year or period for any customer is always around 1 second, that is, the result of the search is immediate. Besides that, the solution was developed with robust algorithms that run in small servers, not demanding machines with powerful processors and several gigabytes of RAM. Therefore, the software s architecture is simple which results in low maintenance and implantation costs. This is combined with a high compression rate which helps to reduce costs even further, in terms of storage, along time as its application will not demand the hard disc frequently. The whole historic data of XDR s, no matter the number of years stored, is always online. The management of more recent data (or, older data) with a certain type of compression (or, even without compression) is not needed. All the data is compressed and ready to be researched independently of the period. 03

4 Architecture overview Mediation XdrWeb FTP Transfer Fixed Voice GSM MMS SMS GPRS Search and display. XdrStorer 1 - Compress 2 - Order and Sort 3 - Validate Duplicity 4 - Index 5 - Transfer to Storage The Xdr system is formed by three components: XdrSender XdrStorer XdrWeb Mediation is typically the main interface in which Xdr works and it will supply the several types of files required to be stored. Modules XdrSender The XdrSender software is the one that searches directly in the compacted data (in the Application Server itself or in Storages) bringing it to the Client station and then processing the de-compressing process in the local station. So, it does not carry out the de-compacting process in the server that stores the files. Security: The system does the management of the users (including the Administrator) and of the researched data with a simple and intuitive interface which includes: Login validation. Search of results and demonstration of the values by profile (some of the users may see all the fields, others may not). Trace reports by users including researches carried out and criteria used. 04

5 Módulos XdrStorer The XdrStorer program is the one that will effectively compress and handle these files transferred and converted into a single layout bye the LOADER. After compressing the file, it will be stored in the same Application Server or in Storages. This program not only works on the compressing and storage but also edit these files. This process is divided into three stages: Stage 1: In this stage, the program only compresses and transfers the files to the destination directory. Stage 2: In this stage, the program picks up the compressed files and sorts out the clients inside the file in order to speed up the searches. All the process done in these stages is logged and may be evaluated and printed. Besides that, the system may issue alert signals if there is any problem in one of the stages giving a detailed report of what happened. Undup: In this stage, the program validates duplicities (the duplicity rule is composed and supplied by the operator company), and organizes the blocks (it is usually done overnight). But even having this option activated, the xdrsearch evaluates online if there are duplicated CDRs and does not display them when found. XdrWeb The XdrWeb program will receive all of these files, validate them according to the layout of the file type and transform them into a single layout. Next, it will transfer these files to the Application Server. The LOADER program is able to process simultaneously several files by executing in parallel. 05

6 CONCLUSION The Xdr software was developed in order to meet telecom companies requirements for data retention. It is a solution that offers unequalled performance with incomparable compression rate. It also demands little hardware installation and execution investment. It is definitely the right and complete solution that will achieve the desired goals and meet requirements at once. Rua DOS Timbiras, th floor Belo horizonte - Minas Gerais - Brasil [email protected] tel.: