The Effect of Transient Detection Errors on RF Fingerprint Classification Performance

The Effect of Transient Detection Errors on RF Fingerprint Classification Performance MEMDUH KÖSE SELÇUK TAŞCIOĞLU ZİYA TELATAR Computer Sciences Research and Electrical and Electronics Engineering Department Application Center Faculty of Engineering Ahi Evran University Ankara University TURKEY TURKEY memduh.kose@ahievran.edu.tr, selcuk.tascioglu@eng.ankara.edu.tr, ziya.telatar@ankara.edu.tr Abstract: - Wireless devices exhibit some unique characteristics depending on their analog circuitry during transmission. These characteristics can be observed from transients of transmitted signals and can be used to generate fingerprints of devices for the purpose of wireless device identification. In this paper, the impact of transient detection errors on an RF fingerprint-based identification system is investigated. Transient detection is carried out by using Bayesian ramp change detector and the performance degradation of this detector due to additive noise is analyzed. Two different RF fingerprints based on instantaneous amplitude responses of transients are used to evaluate the classification performance. Key-Words: -RF fingerprinting, wireless device identification, transient detection 1 Introduction RF fingerprints of wireless devices are generated by exploiting unique characteristics from transmitted signals. The process of analyzing these characteristics to identify wireless devices is called RF fingerprinting. This process has been proposed for security enhancement of wireless networks. RF fingerprinting systems consist of data acquisition, identification signal detection, feature extraction and classification modules. For feature extraction, different signal attributes such as instantaneous amplitude, phase and frequency have been used and several techniques have been employed in time [1], spectral [2] and wavelet domain [3]. Features can be obtained from different parts of the acquired radio signals, such as turn-on transients, RF burst signals, preambles, etc. When the related signal part is obtained, feature extraction process is carried out and RF fingerprints are created from these features. Finally, a classifier is employed to distinguish the fingerprints of different devices. A survey of RF fingerprinting systems can be found in [4]. For an RF fingerprinting system utilizing turn-on transient signals at feature extraction stage, transient detection is a critical task for the reliability of classification system. Several detection schemes have been proposed for the transient detection problem [5]-[10]. In [5], a multifractal segmentation technique based on variance fractal dimension trajectory was used to separate the transient signal from channel noise. It was shown that multifractal characteristics of the channel noise and the transient are different and the location of the change in the fractal trajectory can be detected by setting a threshold. One of the approaches to remove the threshold requirement of this method was proposed by [6], in which the change in the fractal trajectory was detected using a Bayesian change-point detection algorithm. In [7], a transient detection method based on the instantaneous phase responses was proposed. In this method, transient starting point was estimated by using the characteristic differences of phase variance of transient signal and noise. In [9], an algorithm based on mean change point detection was proposed for the detection of transient of WiFi signals. The proposed method was analyzed in terms of complexity and accuracy and was compared with the methods in [5], [6], and [7]. In a recent work by Yuan et al., a transient detection scheme utilizing the complexity difference of noise and transient signals was proposed [10]. The complexity of signals was measured by using permutation entropy trajectory in which a generalized likelihood ratio test detector was employed to find the start of the transient. Inaccurate detection of transient signals causes to loose distinctive features, and therefore the classification performance degrades. So far, the effects of transient detection errors on RF fingerprinting classification system have been analyzed for different transient detection methods. For example, in [2], the impact of noise on transient detection schemes using variance trajectory of ISBN: 978-1-61804-306-1 89

instantaneous amplitude and instantaneous phase responses was investigated. The impact of amplitudebased and phase-based transient detection errors on classification performance is examined at different SNR levels. It was shown that the performance of the variance trajectory method using instantaneous phase responses is unacceptable for IEEE 802.11a signals even at high SNRs. Klein et al. performed a sensitivity analysis for burst detection and fingerprint classification method at varying SNR conditions [11]. The effect of burst transient detection error on classification performance was examined for a multiple discriminant analysis-maximum likelihood classifier. The analysis was carried out for two transient detection approaches in [6] and [7]. It was obtained that the variance trajectory method [7] is superior to the fractal-bayesian step change detector [6] and the results of the variance trajectory method are consistent with the perfect burst detection performance for high SNR levels. In order to improve low SNR performance of variance trajectory detector, a dual-tree complex wavelet transform denoising process was introduced in [12]. In the same study, it was demonstrated through experimental results that introduced denoising process improves both transient detection and classification performance. The impact of burst detection error on classification performance was analyzed for time domain and wavelet domain fingerprints in [3]. Classification performance of wavelet domain fingerprints were obtained to be better than that of time domain fingerprints in case of inaccurate detection of transient. In the same study, it was reported that burst detection error may be occurred due to the operation of the equipment in non-ideal environments or dissimilarity of the equipment used for capturing training and test data. In order to improve the classification performance of a transmitter identification system in noisy channels, a method based on noise injection into the training data of the classifier was proposed in [13]. In this method, training data was contaminated with the noise whose amount was determined by the estimated SNR of the test signal. Classification errors due to the effect of additive channel noise on transient detection and feature extraction stages were considered together for a probabilistic neural network classifier. Performance improvement was demonstrated experimentally using data collected from VHF transmitters. 2 Bayesian Ramp Change Detection Transient detection can be performed using instantaneous amplitude responses of waveforms collected from wireless devices. For a complex valued baseband signal, instantaneous amplitude response is defined as 2 2 A( i) I ( i) Q ( i) (1) where Ii () and Qi () stand for in-phase and quadrature components of the sampled signal at time instant, respectively. In Fig.1, an instantaneous amplitude response of a transmitted signal captured from a WiFi device is shown. As seen from this figure, it is not easy to separate the transient signal from the channel noise exactly due to non-stationary character of transient. i Fig.1. Instantaneous amplitude response of a transmitted signal captured from a WiFi device Üreten and Serinken have proposed a method called Bayesian ramp change detection for transient detection of WiFi signals using instantaneous amplitudes [8]. In this method, transient detection problem is considered as a change-point detection problem. Instantaneous amplitude response is modeled as e( i), if 1 i m Ai () ( i m) e( i), if m i N (2) where m is the change-point, is mean of data before change-point, N is data length, is the ramp slope, e is additive white Gaussian noise. Maximum a posteriori estimate for the change-point is obtained by calculating the posterior probability density with uniform prior information assumption. As an advantage, this method requires neither channel noise ISBN: 978-1-61804-306-1 90

variance knowledge nor the model parameters and. For the details of the algorithm, see [8], in which it was demonstrated that Bayesian ramp change detector has high accuracy for IEEE 802.11b signals at high SNR levels. 3 Bayesian Ramp Change Detector Performance at Low SNR The performance of the Bayesian ramp change detector was evaluated using data collected from eight different IEEE 802.11b devices. Data set contains one hundred transmissions from each device. In order to measure low SNR performance of the detector, SNRs of the collected transients were changed by adding noise. For a fixed SNR level, recorded channel noise signals were added to the collected transients to set the desired SNR level, and then starting points of transients were estimated. This process was repeated for one hundred recorded noise signals. Estimation error for transient starting point is defined as ˆm E mˆ m (3) where and m are the estimated and the actual values of the start of the transient, respectively. Actual starting locations of transients were determined visually from the instantaneous amplitude responses of the collected transient signals. Mean squared error was calculated over 800 transients, each of which were contaminated by 100 different noise signals. MSE values of the Bayesian ramp change detector at different SNR levels is plotted in Fig.2, showing that the decrease in SNR causes an increase in MSE. When the SNR level is less than 10 db, MSE increases dramatically. Fig.2. Transient detection performance of Bayesian ramp change detector at different SNR levels Fig.3. Histogram of transient starting point estimation errors for Bayesian ramp change detector The histogram of the transient starting point estimation errors is demonstrated in Fig.3. Histogram of errors concentrates around zeros at high SNR levels whereas it spreads out as the SNR decreases. This figure also shows that an increasing positive bias in the estimate of transient starting point is induced when the SNR is less than 10 db. 4 The Effect of Transient Detection Errors on Classification Performance Classification is performed using a probabilistic neural network (PNN). After selecting training and test sets randomly, RF fingerprints are generated for these sets. Two different RF fingerprints are employed to evaluate classification performance. The first fingerprint is taken as the instantaneous amplitudes (IA) of transients and the second fingerprint is formed from the features obtained by principal component analysis (PCA) [1]. PCA is a widely used dimension reduction and feature extraction method which projects the data onto a lower dimensional space so that the projected data have maximum variance [14]. The directions of this new space are defined as the eigenvectors of the data covariance matrix, which are called principal components. The number of principal components is determined by different approaches. In a classification problem, this number may be chosen so as to achieve the best classification performance. In [1], it was shown that classification performance of RF fingerprints for WiFi devices remains the same when five principal components of instantaneous amplitude responses are used rather than using all amplitude values. Therefore in this study, we generated RF fingerprints using five principal components. ISBN: 978-1-61804-306-1 91

Additive noise affects the extracted features as well as the estimations of transient starting points. In this work, the aim is to evaluate the effect of transient detection errors on classification performance. Therefore the transients from which features will be extracted are taken as the signals prior to noise contamination. The effect of transient detection errors on PNN classification performance was analyzed experimentally using collected 802.11b transient signals. 20% of all transients from each transmitter were used as training set and remaining transients were used as test set. Training and test sets were selected randomly in each trial. 30 Monte Carlo trials were performed, in each of the trial 30 different recorded noise signals were added to the test signals. Therefore a total of 900 classification results were obtained at each SNR level. Fig.4 shows mean of these classification results for each SNR. This figure demonstrates that correct classification rate decreases as the SNR decreases. Classification accuracy goes below 0.8 when the SNR drops to about 10 db. This figure also shows that the effect of transient detection errors has the same effect on the classification performance of instantaneous amplitude and PCA features for all SNR values in the interval [5, 20] db. Histograms of correct classification rates of PCA features for the case that transient detection is performed at two different SNR levels are given in Fig.5. In order to visualize the effect of transient detection errors on the extracted features, two of the principal components obtained from test transients for the case that transient detection is performed at 20 db and 8 db are shown in Fig.6. This figure shows that features spread out and overlap in the feature space when the transient starting points are incorrectly estimated. Note that only two of the five features are given in this figure to partially visualize the effect of transient detection errors on the separability of PCA features. Overall classification performance cannot be observed from this figure. Fig.4. The effect of detection of transients at different SNR levels on classification performance (a) (b) Fig.5. Classification performance of PCA features for the case that transient detection is performed at two different SNR levels. Fig.6. Two principal components extracted from test transients for the case that transient detection is performed at (a) 20 db and (b) 8 db ISBN: 978-1-61804-306-1 92

5 Conclusion In this paper, the performance of Bayesian ramp change transient detector is evaluated under varying SNR conditions. It is shown experimentally, using collected WiFi signals, that detection performance of the detector degrades significantly when the transient SNR is below 10 db. The effect of transient detection errors on the classification performance of RF fingerprinting system is evaluated by using a probabilistic neural network classifier. Classification performance degradation with increasing transient detection error is shown for two different RF fingerprints based on instantaneous amplitude responses. References: [1] O. Ureten and N. Serinken, Wireless security through RF fingerprinting, Canadian Journal of Electrical and Computer Engineering, Vol. 32, No.1, 2007, pp. 27-33. [2] W.C. Suski, M.A. Temple, M.J. Mendenhall, R.F. Mills, Using spectral fingerprints to improve wireless network security, IEEE Global Telecommunications Conference (GLOBECOM), 2008, pp. 1-5. [3] R.W. Klein, M.A. Temple, M.J. Mendenhall, Application of wavelet-based RF fingerprinting to enhance wireless network security, Journal of Communications and Networks, Vol. 11, No.6, 2009, pp. 544-555. [4] B. Danev, D. Zanetti, S. Capkun, On physicallayer identification of wireless devices, ACM Computing Surveys,Vol.45,No.6, 2012, pp.1-29. [5] D. Shaw and W. Kinsner, Multifractal modeling of radio transmitter transients for clasification, WESCANEX 97: Communications, Power and Computing, 1997, pp. 306-312. [6] O. Ureten and N. Serinken, Detection of radio transmitter turn-on transients, Electronics Letters, Vol. 35, No.23, 1999, pp. 1996-1997. [7] J. Hall, M. Barbeau, E. Kranakis, Detection of transient in radio frequency fingerprinting using signal phase, IASTED International Conference on Wireless and Optical Communications, 2003, pp. 13-18. [8] O. Ureten and N. Serinken, Bayesian detection of Wi-Fi transmitter RF fingerprints, Electronics Letters, Vol. 41, No.6, 2005, pp. 373-374. [9] L. Huang, M. Gao, C. Zhao, X. Wu, Detection of Wi-Fi transmitter transients using statistical method, International Conference on Signal Processing, Communication and Computing (ICSPCC), 2013, pp.1-5. [10] Y.J. Yuan, X. Wang, Z.T. Huang, Z.C. Sha, Detection of radio transient signal based on permutation entropy and GLRT, Wireless Personal Communications, Vol. 82, No.2, 2015, pp. 1047-1057. [11] R.W. Klein, M.A. Temple, M.J. Mendenhall, D.R. Reising, Sensitivity analysis of burst detection and RF fingerprinting classification performance, International Conference on Communications (ICC), 2009, pp. 1-5. [12] R.W. Klein, M.A. Temple, M.J. Mendenhall, Application of wavelet denoising to improve OFDM-based signal detection and classification, Security and Communication Networks, Vol. 3, No.1, 2010, pp. 71-82. [13] O.H. Tekbas, O. Ureten, N. Serinken, Improvement of transmitter identification system for low SNR transients, Electronics Letters, Vol. 40, No.3, 2004, pp. 182-183. [14] C.M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006. ISBN: 978-1-61804-306-1 93