An Informix TimeSeries based Telco Data Retention Solution: Lessons Learned

Transcription

1 An Informix TimeSeries based Telco Data Retention Solution: Lessons Learned Alexander Koerner IBM Germany (On behalf of Cedros Gesellschaft für Datenverarbeitung mbh, Germany) 1

2 Alexander Celebrating 25 Years of Informix InfoWorld, Nov 13th, 1989 ( 2

3 My Project An Informix TimeSeries, REST API, JSON & Raspberry Pi 2 based Weather Station... 3

4 Acknowledgements and Disclaimers Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They areprovided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warrantyof any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBMor its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. All customer examples described are presented as illustrations of how those customers have used IBM products and the results theymay have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Some material in this presentation is Copyright IBM Corporation All rights reserved. U.S. Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo, ibm.comare trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or TM), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Other company, product, or service names may be trademarks or service marks of others. 4

5 Agenda Legal Background cedros An Overview The cedros Telco Data Retention Solution Informix Time Series for CDRs Implementation Specifics Lessons Learned Summary 5

6 Telco Data Retention: Legal Background Europe On 15 March 2006, the European Union adopted the Data Retention Directive. It requires Member States to ensure that communications providers retain the necessary data as specified in the Directive for a period of between 6 months and 2 years in order to... Trace and identify the source of a communication Trace and identify the destination of a communication Identify the date, time, and duration of a communication Identify the type of communication Identify the communication device Identify the location of mobile communication equipment The Directive covers fixed telephony, mobile telephony, Internet access, , and VoIP On 8 April 2014, the Court of Justice of the European Union declared the Directive 2006/24/EC invalid for violating fundamental rights The Council's Legal Services have been reported to have stated in closed session that paragraph 59 of the European Court of Justice's ruling "suggests that general and blanket data retention is no longer possible" Source: Wikipedia 6

7 Telco Data Retention: Legal Background United States The National Security Agency (NSA) commonly records Internet metadata for the whole planet for up to a year in its MARINA database, where it is used for pattern-of-life analysis. U.S. persons are not exempt because metadata are not considered data under US law. Its equivalent for phone records is MAINWAY. The NSA records SMS and similar text messages worldwide through DISHFIRE. Various United States agencies leverage the (voluntary) data retention practised by many U.S. commercial organizations through programs such as PRISM and MUSCULAR The United States does not have any Internet Service Provider (ISP) mandatory data retention laws similar to the European Data Retention Directive. All attempts to create mandatory retention legislation have failed. Source: Wikipedia 7

8 Agenda Legal Background cedros An Overview The cedros Telco Data Retention Solution Informix Time Series for CDRs Implementation Specifics Lessons Learned Summary 8

9 cedros An Overview cedros isoneoftheoldest and biggest IBM Premier Business Partners in Germany and authorized for resale in all software families cedros develops information and communication technology that is characterized by high creation of value for customers. Efficiency and investment security are of crucial importance in this context Their primary competencies are in four areas: Enterprise Software Solutions Telco Solutions & Services Software Infrastructure & Services IBM Software sales 9

10 cedros Telco Solutions Legal Compliance Telecommunications operators or service providers are obligated by law to setup technical systems in order to monitor their infrastructures and, when asked to do so, to provide information on the stored data or communication content Cedros develops solutions that realize both the technical systems and the information process Efficiency, convenience and data protection are of central importance along the way 10

11 Agenda Legal Background cedros An Overview The cedros Telco Data Retention Solution Informix Time Series for CDRs Implementation Specifics Lessons Learned Summary 11

12 Data Retention Requirements According ETSI TS V ( ) Telephony / Mobile / VoIP Originating(A) and destination (B) number/id, forwarding (C) number Start/End time, time zone for the call; for VoIP: IP address for A/B/C For mobile: IMEI, IMSI, Cell-ID (for all involved subscribers) Call type/service (Voice, Fax, Data, SMS, MMS, GPRS, LTE etc.) Prepaid: First activation, cell id; Internet Access Login-Name, assigned IP-Address (Radius/DHCP) Line/Port identifier, Connection/Service Type Start-/End of the session Service in/out: Subscriber Id, From/To/CC/BCC, Message-Id, Date/Time, Servers, IP-Addresses Mailbox Access: Subscriber Id, server, IP addresse of the client 12

13 cedros Data Retention Solution Data Storage Server: c.-drs implements data retention according to legal requirements uses IBM Informix Version 12 for the data store utilizes the Informix Time Series feature provides an Web UI for simple retrieval process Information Process/Workflow: c.-ais connects to all relevant backend systems (DR store, invetory data, ERP, LI system etc.) implements the complete workflow including data processing/formatting, invoicing etc. runs or IBM Lotus Notes or as Web UI 13

14 Sample Infrastructure Network element Logfiles Invetory Data Approval by clerk Radius Mediation (CDR) c.-drs Informix with Time Series: Retention Data Case Authorities Case History ERP- System Invoice 14

15 A Customer Example A Swiss telecommunications provider Requirements based on Swiss data retention laws Implemention close to the European Telecommunications Standards Institute s (ETSI) standards CDR processing Input format is CSV via SFTP TSL-loaded into Informix Query definition via Web Interface Results sets are generated in XML About million CDRs/day Data retention period: 180 days Database size: 6 TB 15

16 Agenda Legal Background cedros An Overview The cedros Telco Data Retention Solution Informix Time Series for CDRs Implementation Specifics Lessons Learned Summary 16

17 Informix Time Series forcdrs 1/2 Informix providesan optimizedstorageforthe Retention Data: Subscriber Idbuildsthe primarykey, CDR dataisstoredastime series The Time Series Loaderprovidesan optimized waytoloadhugeamountsoftime seriesdata Data canberetrievedbasedon virtualtables (lookslike classic SQL table) High scaleabiliy, runson Linux 17

18 Informix Time Series forcdrs 2/2 CDRs access requirements are typically based on the primary key(s) Provide me all communication activities for a given subscriber id or a set of subscriber ids with an optional date range The Informix Time Series rolling windows container concept is the perfect functionality to fulfill legal data retention period requirements JSON based Time Series can be utilized for flexible CDR data structures 18

19 Agenda Legal Background cedros An Overview The cedros Telco Data Retention Solution Informix Time Series for CDRs Implementation Specifics Lessons Learned Summary 19

20 Implementation Specifics The current cedros c.-drs server is based on Informix FC5 We did run into some TSL resource issues in FC4W1 which have been addressed in FC5 cedros wanted to utilize some of the new FC5 TSL features CDR records can be provided in different formats (e.g. XML) and are converted into a suitable TSL format The original records are archived to due legal requirements 20

21 Structured TimeSeries vs. JSON TS Each class of CDRs might has its own elements Telephony CDRs Internet Access CDRs CDRs One TimeSeries for each CDR class or one TimeSeries for all kind of CDRs classes? A JSON TimeSeries provides complete data structure flexibility Slight storage and processing (parsing) overhead Maximum TS JSON element size (12.10.xC5): 4 kb / JSON document 21

22 Example for a CDR JSON TimeSeries create row type cdr_data_t ( cdr_event datetime year to fraction(5), cdr_element bson); create table cdr_ _ts ( subscriber_adr varchar(254), msgstore_id varchar(32), subscriber_id varchar(64), cdr_recs timeseries(cdr_data_t), primary key (subscriber_adr, msgstore_id, subscriber_id) ); execute procedure TSCreateVirtualTab ( cdr_ _v', cdr_ _ts', 'origin( :00: ), calendar(cdr_1sec), container(tscontainerpoolroundrobin(cdr_cont_pool)), threshold(0),irregular', 'putelem,scan_discreet', cdr_recs'); insert into cdr_ _v values ( akoerner@de.ibm.com, MSGSTORE001, Alexander Koerner, :30:00 ::DATETIME YEAR TO SECOND, { "seq_number":1, "ts_import":" :55:00", "client_id":"test client_id", "other_ _adr":"andreasw@de.ibm.com", "message_id":"message_id_01_02_03", "op_status":"s", "server_id": "server_id 001", "protocol":"i", "srcsrv_name":"sender_server_001", "dstsrv_name":"receiver_server_001", "operation":"t" } ::JSON 22

23 Rolling Window Containers CDR data retention periods are defined by legal regulations European Union guidance: between 6 and 24 months Italy: ISP data needs to be stored for 12 months and telephony data for 24 months Switzerland: ISP data needs to be retained for 6 months (180 days) Germany (draft): 4 weeks (cell data) 10 weeks (phone numbers, call details, SMS/MMS time stamps for mobile phone telephony/messaging), 10 weeks for Internet ISP data Deleting single time series elements after the retention period is not very efficient (e.g. via the DelElem() or the DelRange() TS functions) The TSContainerPurge() TS function might be better for bulk deletes The Rolling Window Container feature (introduced with Informix xC1)provides a fast and elegant way of purging old ( aged ) time series data automatically 23

24 Rolling Window Container Example execute procedure TSContainerCreate ( 'cdr_cont1', 'cdr_cont1_dbs1, cdr_cont1_dbs2, cdr_cont1_dbs3', 'cdr_data_t', , , ' :00: '::datetime year to fraction(5), 'day', 180, 1, NULL, 1, 2048, 512 ); execute procedure TSContainerCreate ( 'cdr_cont2', 'cdr_cont2_dbs1, cdr_cont2_dbs2, cdr_cont2_dbs3', 'cdr_data_t', , , ' :00: '::datetime year to fraction(5), 'day', 180, 1, NULL, 1, 2048, 512 ); execute procedure TSContainerSetPool ('cdr_cont1', 'cdr_cont_pool'); execute procedure TSContainerSetPool ('cdr_cont2', 'cdr_cont_pool'); 24

25 Resource Efficiency Carefully choose appropriate page sizes: CDRs come in large amounts and might contain many elements! One can store up to 254 TS elements/logical page Each TS has its own sets of pages Example: One CDR is on average 200 bytes. There are about 10 CDRs per TS per day 10 x 200 bytes = 2000 bytes 2 kb page size For JSON CDR Time Series consider using short JSON element identifiers Max JSON document size in a time series 4 kb Less JSON identifier length less disk storage If you choose a too large page size......you will have wasted space on disk...you might run into some buffer pool memory size issues (due to unused memory page space) 25

26 CDR Load Performance The VTI interface is suitable for small batches of CDRs and/or continous loading of single records Use the TS Loader API for large amounts of CDR data Obey the obvious recommendations for disk I/O optimizations (e.g. I/O seperation of logical logs chunks from the dbspaces chunks) Fragment the TS base table and distribute the TS across multiple containers Consider using parallel TSL sessions Utilize the new reduced TSL message logging capabilties in xC5 To simplify the loading process, you might want to use the new.xc5 TSL_SetNewTS() function to automaticall create a new TS in the base table if that TS doesn t exist 26

27 CDR TSL Example execute function TSL_Init('cdr_ _ts','cdr_recs',3,4,NULL,'%Y-%m-%d %H:%M:%S','/tmp/cdr_rejected.log',NULL); execute function TSL_SetLogMode ('cdr_ _ts cdr_recs',1,2,'/tmp/cdr_error.log'); execute function TSL_SetNewTS('cdr_ _ts cdr_recs', 'origin( :00: ), calendar(cdr_1sec), TSContainerPoolRoundRobin(cdr_cont_pool), threshold(0), irregular', 1); execute function TSL_Put ('cdr_ _ts cdr_recs',filetoclob('/tmp/test.mail','server')); begin work; execute function TSL_FlushAll('cdr_ _ts cdr_recs'); commit work; execute function TSL_SessionClose('cdr_ _ts cdr_recs'); execute procedure TSL_Shutdown('cdr_ _ts cdr_recs'); 27

28 Agenda Legal Background cedros An Overview The cedros Telco Data Retention Solution Informix Time Series for CDRs Implementation Specifics Lessons Learned Summary 28

29 Lessionslearned 1/2 High Data volume and complexity of data requires a careful design ofthedatamodel Choose the optimal primary key(s) to avoid too small time series Carefully choose the optimal page size and disk location for time series containers Consider the usage of rolling window containers for legal requirements and simplified data housekeeping (time series data deletion) Use latest Informix TS loader features Informix xC5 provides some very helpful new TSL capabilities and better resource usage Automatic creation of missing time series Support for JSON time series Load time series data from an external file 29

30 Lessionslearned 2/2 Data volumecanvaryextremly, esp. Events createdby Service (mobile IMAP!) Telcosdon tknowtheirdatastructures (source, volume, quality) in detail Beawareofdifferent numberingplans(0, 0049, 49 ) anddataqualityproblems 30

31 Questions? 31

32 Alexander Koerner vcard 32