MONITORING METHODOLOGY TO ASSESS THE PERFORMANCE OF GSM NETWORKS

Electronc Communcatons Commttee (ECC) wthn the European Conference of Postal and Telecommuncatons Admnstratons (CEPT) MONITORING METHODOLOGY TO ASSESS THE PERFORMANCE OF GSM NETWORKS Athens, February 2008

0 EXECUTIVE SUMMARY Ths report presents a procedure to assess the performance of GSM networks. It s based on the procedure used by the Belgan admnstraton however the general concepts n ths document apply to every GSM network. The prncple of the methodology s to check whether the operators requrements, whch are contaned n ther lcence, are beng met. It must be noted that there may be partcular requrements on the performance to be reached by natonal GSM networks. The operators requrements presented here concern the degree of coverage relatng to the terrtory, the populaton and the hghways as well as the blockng call rate and the droppng call rate. As far as the coverage rates are concerned, the prncple of the method conssts n askng operators to provde ther coverage fles and checkng the relablty of ths data by means of measurements. Ths procedure makes t possble to lmt the sample of measurements to verfy the operator data.. As far as the blockng and droppng rates are concerned, specfc tests are carred out. The prncple of these tests s to carry out a sequence of calls and each call s scored dependng on how t took place. A mnmum number of calls has to be carred out and the blockng and droppng rates are deducted from the statstcs on ths score. The general prncple of the montorng methodology s to assgn an ndex to the dfferent tests carred out and to combne these dfferent ndexes nto a general ndex.

Table of contents 0 EXECUTIVE SUMMARY...2 1 INTRODUCTION...4 2 SIGNIFICANT PARAMETERS TO ASSESS THE PERFORMANCE OF GSM NETWORKS...4 3 PRINCIPLE OF THE NETWORK ASSESSMENT PROCEDURE...4 4 EQUIPMENT TO MEASURE THE NETWORK AND ASSESS RESULTS...4 4.1 DATA PROCESSING SOFTWARE...4 4.2 MEASURING SECTION...5 5 EXAMPLE OF LICENCE REQUIREMENTS...6 6 MEASUREMENT OF THE COVERAGE RATE...6 6.1 DECISION CRITERION...6 6.2 ANALYSIS OF THE OPERATOR S COVERAGE AREA...7 6.2.1 Coverage predcton fle...7 6.2.2 Treatment of the theoretcal coverage fles...7 6.2.3 Calculaton of the theoretcal coverage ndexes...8 6.3 MEASUREMENTS ON SITE...8 6.3.1 Choce of the test routes...8 6.3.2 Carryng out of the measurements on ste...8 6.3.3 Assessment of the measurements along the route...9 6.4 CALCULATION OF THE COVERAGE INDEXES OF THE TERRITORY, THE POPULATION AND THE HIGHWAYS AND OF THE COVERAGE INDEX OF THE NETWORK...10 7 MEASUREMENT OF THE BLOCKING RATE...11 7.1 PRINCIPLE OF THE TEST...11 7.2 NUMBER AND LOCATION OF THE TESTS...11 7.3 TEST TIME...11 7.4 TEST PROCEDURE...11 7.5 CALCULATION OF THE BLOCKING RATE INDEX...12 8 MEASUREMENT OF THE DROPPING RATE...12 8.1 PRINCIPLE OF THE TEST...12 8.2 NUMBER AND LOCATION OF THE TESTS...12 8.3 TEST TIME...12 8.4 TEST PROCEDURE...12 8.5 CALCULATION OF THE DROPPING RATE INDEX...13 9 GENERAL QUALITY INDEX...13 10 REFERENCES...14

1 INTRODUCTION Ths document presents frst the sgnfcant parameters to assess the performance of a GSM network, the prncple of the assessment procedure and the equpment necessary to carry out a measurng campagn. After these three tems of a rather general nature, an example of a specfc objectve to be met by an operator s presented and the three followng tems tackle n detal the methodology of determnng the coverage, blockng and droppng rates. Fnally the last tem concerns the calculaton of the general ndex and enables us to decde whether or not the operator has honoured ther commtments. 2 SIGNIFICANT PARAMETERS TO ASSESS THE PERFORMANCE OF GSM NETWORKS Three parameters are sgnfcant to assess the performance of GSM networks: the coverage, the blockng rate and the droppng rate. Coverage The coverage of GSM-networks n a country can be expressed n percentage of coverage relatng to the terrtory, the populaton or both. Moreover specfc oblgatons such as coverage of some trunk roads can be set. The meanng of covered and uncovered areas has to be clearly defned wth a common understandng by all partes nvolved. Unless otherwse stated n the lcense condtons, GSM networks should be consdered as a whole and the coverage should be measured on all relevant frequency bands smultaneously. Blockng rate The blockng rate s the percentage of blocked call attempts whch are caused specfcally by the operator's network aganst the sum of good and blocked calls. Droppng rate The droppng rate s the percentage of calls that are dropped durng the call perod whch are caused specfcally by the operator's network aganst the sum of good and dropped calls. Other parameters Other parameters such as audo qualty can be assessed, however, they are not consdered n ths report. 3 PRINCIPLE OF THE NETWORK ASSESSMENT PROCEDURE To assess the performance of a GSM s network the operator's oblgatons have to frst be defned. The assessment procedure wll then am at checkng f the operator s oblgatons are met. Ths assessment s carred out by analysng the suppled data and the results of practcal measurements. If more than one network has to be assessed the measurements should be performed smultaneously. All tests carred out are tagged wth ndexes whch are combned nto a general ndex. If the general ndex exceeds 1 the network can be consdered as fulfllng ts oblgatons. 4 EQUIPMENT TO MEASURE THE NETWORK AND ASSESS RESULTS For the assessment, data processng software and approprate measurng equpment are requred. 4.1 Data processng software A GIS (Geographc Informaton System) applcaton s used whch makes t possble to handle and to process the dfferent nformaton on network coverage.

4.2 Measurng secton The block dagram of the measurng equpment whch s nstalled n a measurement vehcle can be seen below. G S M G S M G S M GPS Geo locatng devce P.C. Compass s Odometer The measurement equpment comprses: - several test moble transcevers to test all operators smultaneously - a laptop equpped wth a measurement software - a postonng system The moble transcevers used are specally developed for ths type of tests and have an nterface for communcatng wth the laptop. The software of the measurng equpment - controls the measurement mobles - records mportant sgnallng data exchanged between the moble and the base staton; - receves the poston nformaton from the postonng system; - stores raw data to hard dsk - processes and dsplays the desred data; - exports the requred data n the defned format. The postonng system conssts of a GPS-devce. A dead reckonng system that enables contnued recordng of measurng pont coordnates f GPS coverage s lost can also be added. Ths optonal system s equpped wth an electronc compass to defne the drecton and an odometer to determne the dstance travelled. Ths allows extrapolaton of the vehcle coordnates from the last receved poston should the GPS sgnal be lost. The measurng antennas are mounted on the roof of the vehcle, at a heght between 1.5 and 3 meters. The measurng system ncludng the postonng system has to be calbrated before use.

5 EXAMPLE OF LICENCE REQUIREMENTS Three objectves have been defned for the coverage rate: - Coverage of the terrtory: obj cov, ter (e.g. 85%), weghtng factor: P ter (e.g. 35%) - Coverage of the populaton: obj cov, pop (e.g. 98%), weghtng factor: P pop (e.g. 35%) - Coverage of the hghways: obj cov, hw (e.g. 100%), weghtng factor: P hw (e.g. 30%) The weghtng factor wll be descrbed n secton 6.4. Further objectves have also been defned for the blockng rate (e.g. 3 %) and the droppng rate (e.g. 2%). 6 MEASUREMENT OF THE COVERAGE RATE The coverage assessment s based on the rules set for the GSM operator such as the coverage declaraton by the operator. After analyss of ths declaraton, actual measurements wll be made to ensure that t corresponds to realty. Ths method makes t possble to lmt the number of measurement routes. 6.1 Decson crteron Frst of all the crteron to decde f a locaton s covered or not has to be defned. For that purpose we base our judgment on two parameters - RxLev and RxQual - whch are measured by the test GSM every 470 ms. These parameters are representatve of the receved level and the qualty of the call. Level Every GSM moble measures the parameter RxLev. Durng the call ths parameter corresponds to the power level receved by the moble on the transmsson channel. The measurement scale vares from - 110 dbm to - 47 dbm although the moble actually reports another value (step) whch vares from 0 to 63 wth 0 correspondng to -110 dbm and 63 correspondng to 47 dbm or greater. Two parameters can be measured: RxLevFull and RxLevSub (although the generc term RxLev s used n ths report). RxLevSub s more approprate for ths type of measurements. The report must clearly state whch parameter was measured. Qualty Every GSM n the process of a call measures the parameter RxQual. Ths parameter vares from 0 (best) to 7 (worst). Two parameters can be measured: RxQualFull and RxQualSub (although the generc term RxQual s used n ths report). RxQualSub s more approprate for ths type of measurements. The report must clearly state whch parameter was measured. Decson threshold The decson threshold s used to establsh f a locaton s covered or not. Every admnstraton s free to defne ts decson thresholds. For nformaton these thresholds can be for an outdoor coverage: RxLev - 92 dbm [ RxLev 18 (step) ] RxQual 5 Lnk between the power measured and the electrc feld strength The measurement of RxLev ndrectly enables us to assess the feld radated on the measurement frequency. The converson factor between the power measured by the GSM and the external feld strength s determned from the followng theoretcal formula: E (dbµv/m) P(dBm) + C E (dbµv/m) P(dBm) + 20 * log f (MHz) + 77,2 (db) - G (db) + P con (db)

where: E electrc feld strength P power measured C converson factor f frequency G sotropc gan of the antenna P con loss at the level of connectons Therefore t s mportant to defne for whch knd of GSM the condton (RxLev - 92 dbm) has to be met. By applyng the GSM standard, we have defned that the measures should theoretcally be carred out by a reference GSM, whch satsfes the equaton: - G (db) + P con (db) 0 The converson factor n the case of a reference GSM, s therefore: C ref,900 20 * log F (942,5) + 77,2 + 0 136,7 C ref,1.800 20 * log F (1842,5) + 77,2 + 0 142,5 (C ref,900 ;C ref,1.800 ) (136,7 ;142,5) The threshold expressed n terms of feld strength corresponds to: E ref,900-92 + 136,7 44,7 dbµv/m E ref,1800-92 + 142,5 50,5 dbµv/m Hence the decson threshold, as far as the level receved s concerned, can therefore be expressed as follows: At the locaton of the measurement, there must be a mnmum feld strength of 44.7 dbµv/m at 900 MHz and of 50.5 dbµv/m at 1800 MHz on the transmsson channel. At ths locaton, the measurement should theoretcally be carred out by a reference GSM that would measure, for these feld values, a level of -92dBm. In practce, the measurng chan s not composed of a reference GSM and t wll be necessary to take ts characterstcs nto account as part of the calbraton of the recevng chan. Ths calbraton results n a set of calbraton factors that are stored n the measurng software and are taken nto account durng the measurements. Ths allows the recevng chan to be consdered deal. 6.2 Analyss of the operator s coverage area 6.2.1 Coverage predcton fle It s necessary that the operators delver all the nformaton needed to evaluate the coverage of ther network. For example, a fle representng the combned coverage of ther networks at 900 MHz and 1800 MHz n a format compatble wth the GIS applcaton used. Ths fle s made up of a grd of squares, for example 50m x 50m, correspondng to the country. For every square the operator has to ndcate f the cell s covered or not. The followng sectons descrbe a way to evaluate the coverage of the operator based on ths assumpton. It must be consdered as an example. 6.2.2 Treatment of the theoretcal coverage fles The frst step conssts n analysng the operator's theoretcal coverage fles. The data processng software calculates the percentage of the terrtory surface covered accordng to the operator s declaraton (D cov,ter ). To calculate the percentage of the populaton covered, reference to the data of the Natonal Insttute of Statstcs of the country should be used. By combnng ths data to the data delvered by the operator the software analysng the data can calculate the percentage of the populaton covered accordng to the declaratons of the operator: D cov,pop. For the calculaton of the percentage of hghways covered accordng to the operator's declaraton (D cov,hw ) the software uses the hghways geographcal data and the coverage declaraton of the operator.

Therefore ths assessment of the operator's coverage fles makes t possble to verfy ther declaraton for the coverage of the surface, the populaton and hghways. 6.2.3 Calculaton of the theoretcal coverage ndexes The ndexes n ths document can be used to assess the dfference between the operator's declaraton and ther oblgatons. The declaraton of the coverage of the terrtory (D cov,ter ) s compared to the objectve defned n the operator's lcence (obj cov, ter ) n order to defne the ndex of the terrtory theoretcal coverage: th, ter D Obj cov, ter cov, ter The ndex of the theoretcal coverage of the populaton s smlarly defned by comparng the coverage declaraton of the populaton (D cov,pop ) to the objectve defned n the operator s lcence (obj cov, pop ). th, pop D Obj cov, pop cov, pop Smlarly the coverage ndex of the hghways s calculated by comparng the coverage declaraton of the operator for hghways (D cov,hw ) wth ther oblgatons (obj cov, hw ). th, hw D Obj cov, hw cov, hw 6.3 Measurements on ste On ste measurements are conducted to verfy the data of the operator. 6.3.1 Choce of the test routes We present below an example of a test route program: Ten test routes have been chosen to check the coverage of the terrtory. The followng crtera are appled for ther choce: - The country s muncpaltes are dvded up nto 5 categores of populaton densty. - Only the muncpaltes covered at least at 10% are retaned. - Two muncpaltes are chosen randomly n each of the categores. - The selected muncpaltes must be separated by at least 30km. - The route of around 100km s defned through and round ths muncpalty. Fve test routes have been chosen for hghways, of a length of about hundred klometers. They are also randomly chosen.. 6.3.2 Carryng out of the measurements on ste Terrtory measurement route The test route frst defned s transferred to the measurng team ether on paper maps or electroncally. So for example the use of the software makes t possble to defne the route on a computer and to download t electroncally to the vehcle's GPS. All operators are smultaneously tested. Every measurng equpment GSM s set to make a contnuous call to a dedcated answerng machne placed on the network of the operator under test. The call enables the montorng of the operator s network. Ths answerng machne must pck up when the call s made and mantan the call as long as t s not dropped by the measurng equpment.

Durng the test all the parameters of the call are regstered (sgnallng data exchanged between the GSM and the network) by the measurng equpment and can be vsualsed by the techncan who checks that the test s progressng well. Havng completed the test drve the recorded data can be processed. The two useful parameters are RxLev and RxQual. The data must be exported n a format compatble wth the GIS applcaton. For example, a 30m x 30m grd s supermposed on the measurement route. The average of the RxLev and RxQual values measured nsde a square are calculated and the averages are assocated wth the center of the square. Ths results n a fle n ASCII format, n whch each lne s composed of the coordnates n lattude/longtude of the centre of the square and of the parameters RxLev and RxQual assocated wth that square. Measurement routes of the hghways These routes are carred out n the same way as the terrtory measurement routes. The exportaton of data s however made by supermposng a grd of 60m x 60m nstead of 30m x 30m on the route. Ths takes account of the hgher speed of the measurement vehcle. 6.3.3 Assessment of the measurements along the route The fles from the measurng equpment are mported to the data processng software. It s necessary to assess the measurng data. For that purpose the measurements made on ste are compared to the declaraton of the operator. Ths comparson results n the defnton of the measured coverage ndexes of whch an example s gven below. Calculaton of the measured ndex for the terrtory measurement routes The 30m x 30m grd s supermposed on the 50m x 50m grd on whch the calculatons are carred out. The average of all the values n a square of 50m x 50m s worked out and assocated wth the centre of ths square. It should be noted that the grd spacng used for the measurements should preferably be the same as used for the coverage predctons however suffcent care has to be taken n order to avod a systematc 'offset' gap between the measured data and the theoretcal data. The followng operatons are carred out on the fle contanng the route referred to on the 50m x 50m grd: - For every square the value 1 s assgned f the decson thresholds are exceeded (n our example, RxLev 18 (step) and RxQual 5), otherwse the value 0 s assgned. - The percentage of the terrtory covered along the route accordng to the measurements s calculated: % mes,ter,. - The percentage of the terrtory covered along the route accordng to the declaraton of the operator s calculated: % th,ter,. - The ndex for the route s calculated as follows: mes, ter, % mes, ter, 0,95 x % th, ter, The factor 0.95 s ntroduced n order to take all measurng uncertantes nto account: f a measured coverage ndex s 95% of the declaratons of the operator, the ndex s worth 1. The coverage ndex measured for all the routes s calculated from these dfferent ndexes by takng nto account a weghtng factor proportonal to the length of the routes: mes, ter x mes, ter, drve tests l drve tests l

Calculaton of the ndex measured for the motorway measurement routes The calculaton s done n a smlar way. The 60m x 60m grd s mported and we deduct the ndex for the route of the percentage of the terrtory covered along the route accordng to the measure (% mes,hw, ) and accordng to the declaraton of the operator (% th,hw, ). mes, hw, % mes, hw, 0,95 x % th, hw, The coverage ndex measured for all the routes s deducted n the same way too: x mes, hw drve tests l mes, hw, drve tests l 6.4 Calculaton of the coverage ndexes of the terrtory, the populaton and the hghways and of the coverage ndex of the network It s necessary to assgn a coverage ndex to the network. Ths ndex wll ndcate to what extent the network meets ts coverage oblgatons. For ths purpose the varous ndexes prevously calculated are combned. Ths relates to the theoretcal coverage ndexes (comparson between the declaraton of the operator and the objectves of ther lcence) and the measured coverage ndexes (comparson between the measurements and the declaraton of the operator). Coverage ndex of the terrtory The coverage ndex of the terrtory s calculated from the theoretcal coverage ndex of the terrtory multpled by the measured coverage ndex of the terrtory. ter th, ter x mes, ter Coverage ndex of the populaton It s mportant to note here that we do not defne any measured ndex of the populaton. Ths s due to the dffculty to calculate the populaton percentages along the measurement routes and the sources of uncertanty t can brng about. The coverage ndex of the populaton s thus calculated from the theoretcal coverage ndex of the populaton by multplyng t by the measured coverage ndex of the terrtory. pop th, pop x mes, ter Coverage ndex of the hghways The coverage ndex of the hghways s calculated from the theoretcal coverage ndex of the hghways and multpled by measured coverage ndex of the hghways. hw th, hw x mes, hw Coverage ndex of the network In order to combne the three sub-ndexes, t s necessary to assgn relatve weghts to them. To that end we use the weght factors that have been defned n the lcence of the operators. P x + P x + cov ter ter pop pop P hw x hw

7 MEASUREMENT OF THE BLOCKING RATE 7.1 Prncple of the test The measurng equpment s set to ntate a seres of successve calls. Every call or call attempt s categorsed dependng on the sgnallng messages exchanged wth the network whch can be: Good : The call was successful. Blocked : Sgnallng messages between the network and the test GSM ndcate, durng the phase of puttng the call through, that the call s mpossble. Dropped : Sgnallng messages between the network and the test GSM ndcate, durng the call, that ths call s dropped. No servce : There s no coverage. The number of calls n each category s regstered by the measurng equpment and the results of the blockng test are calculated on the bass of ths data. Tests are carred out at fxed ponts. There could be other types of tests, for example along the route. 7.2 Number and locaton of the tests We present below an example of chosen tests. In total sx tests are carred out. Two muncpaltes are chosen at random n each of the three groups of hghest populaton densty (accordng to the classfcaton of the country s muncpaltes nto fve groups of populaton densty). Other tests stes can be chosen, such as busness areas wth low populaton densty. The tests measurements are carred out at locatons where RxLev -70 dbm n order to avod blockng calls because of the bad qualty of the rado lnk. 7.3 Test tme Tests should preferably be carred out at peak hours. These peak hours should be defned n co-operaton wth the operators. If the peak hours consderably dffer from one operator to another t mght be mpossble to test the dfferent operators smultaneously. If t s too dffcult to clearly defne peak hours the admnstraton may decde to carry out the tests at the hours of ts choce durng workng days. 7.4 Test procedure The test conssts of the repetton of the followng sequence: 1. Puttng the call through 2. Mantenance of the call for 15 s 3. Termnaton of the call 4. Watng tme of 15 s If the call has been blocked, a 15 s watng tme s observed before any new attempt s made. The call s made to a dedcated answerng machne placed on the operator s network to ensure that any blockng recorded s caused by the operator s network. The software regsters the dfferent outcomes concernng the calls durng the test and the procedure s stopped when between 200 and 220 calls that are qualfed of Good have been made.

The blockng rate for the test s calculated as follows: number ' blocked' calls t bloc, test number ' good' calls + number ' blocked' calls It should be ponted out that n ths calculaton we do not take nto account the calls that could have categores other than Good or Blocked so that the test only concerns the blockng performances. 7.5 Calculaton of the blockng rate ndex In order to assess the result of the test t s necessary to combne the results of the dfferent tests and to defne an ndex for the blockng rate. The blockng rate for all the tests s calculated by weghtng the blockng rates of the dfferent tests dependng on the populaton densty of the muncpaltes where the tests have been carred out: ρ pop, test t bloc tbloc, test tests ρ pop, test tests The blockng ndex ams at comparng the result of the blockng tests (t bloc ) to the objectves set n the lcence (obj bloc ). The followng formula can be used by lmtng the blockng ndex to 1 (see comment n secton 9): bloc tbloc obj 1 bloc 2 x objbloc 8 MEASUREMENT OF THE DROPPING RATE 8.1 Prncple of the test The prncple of the test s dentcal to the prncple used for the measurement of the blockng rate,.e. makng successve calls and categorsng them as 'Good', 'Dropped', 'Blocked' or 'No Servce' (see tem 7.1). The tests are carred out along a route. 8.2 Number and locaton of the tests Below s an example of a chosen test. Fve tests are carred out. The country s dvded nto fve regons. In each regon a route s defned, that must be stuated n an area declared covered by the operator. Ths enables us to avod the dropped calls because of the edge of the coverage area of the operator. If several operators are beng tested smultaneously the route has to be defned n an area covered by all of these operators. 8.3 Test tme The comments made n secton 7.3 also apply. 8.4 Test procedure The followng sequence s carred out: 1. Puttng the call through 2. Mantenance of the call for 2 mnutes 3. Termnaton of the call 4. Watng tme of 15 s

If the call has been dropped, a 15 s watng tme s observed before any new attempt s made. Agan the call s made to a dedcated answerng machne placed on the operator s network to ensure that any droppng recorded s caused by the operator s network. The procedure s stopped when we have made between 100 and 110 calls that are categorsed as Good. The droppng rate for the test s calculated as follows: t drop, test number ' dropped' calls number ' good' calls + number ' dropped' calls It should be ponted out that n ths calculaton we do not take nto account the calls that could have categores other than Good or Dropped so that the test only concerns the droppng performances. 8.5 Calculaton of the droppng rate ndex In order to assess the result of the droppng tests t s necessary to combne the results of the dfferent tests and to defne a droppng ndex. Below an example for these calculatons: The droppng rate for all the tests s calculated by workng out the average of the droppng rates of the dfferent tests. tdrop, test tests tdrop Number of tests The droppng ndex, amed at comparng the results of the measurements (t drop ) wth the objectves of the operator (obj bloc ), can be calculated as follows, by lmtng the droppng ndex to 1 (see tem 9 Comment): drop tdrop obj 1 2 x obj drop drop 9 GENERAL QUALITY INDEX The procedure ams at checkng f the operator s objectves are met. For that purpose all the tests carred out make t possble to defne three ndexes, correspondng to the montorng wth respect of the three objectves of the operator: The coverage ndex of the network (see secton 6): cov The blockng ndex (see secton 7) : bloc The droppng ndex (see secton 8) : drop These three ndexes should be combned n a general ndex. For example the followng formula can be used (the formula may be modfed dependng on the mportance the admnstraton puts on the dfferent objectves): 0,8 x (10 x cov 9) + 0,1 x + 0, 1 x gen If the general ndex s greater 1 the operator s consdered to comply wth ther lcence. Comment on the ndexes presented n ths document The ndexes presented for nformaton n ths document have the followng lmts: cov [ 0 ; > 1 ] bloc [ < 0 ; 1 ] coup [ < 0 ; 1 ] gen [ < 0 ; > 1 ] bloc drop

Wth the ndexes defned n ths way and the calculaton formula of the general ndex ndcated above the operator can make up for a bad result for the droppng or blockng rate (blockng or droppng ndexes lower than 1) thanks to a good result concernng coverage (coverage ndex hgher than 1), but the contrary s not true. If the coverage ndex s lower than 1, the general ndex wll be lower as well because the blockng and droppng rates are lmted to 1. 10 REFERENCES GSM standards as publshed by the European Telecommuncatons Standards Insttute.