Hybrid spam filtering for mobile communication

Transcription

1 coputers & security 29 (2) available at journal hoepage: Hybrid spa filtering for obile counication Ji Won Yoon a, Hyoungshick Ki b, *, Jun Ho Huh c a Statistics Departent, Trinity College Dublin, Ireland b Coputer Laboratory, University of Cabridge, Cabridge, UK c Coputing Laboratory, University of Oxford, UK article info Article history: Received 22 August 29 Received in revised for 7 October 29 Accepted 6 Noveber 29 Keywords: Spa SMS essages Hybrid Content-based filtering Challenge-response Threshold sensitivity proble abstract Spa essages are an increasing threat to obile counication. Several itigation techniques have been proposed, including white and black listing, challenge-response and content-based filtering. However, none are perfect and it akes sense to use a cobination rather than just one. We propose an anti-spa fraework based on the hybrid of contentbased filtering and challenge-response. A essage, that has been classified as uncertain through content-based filtering, is checked further by sending a challenge to the essage sender. An autoated spa generator is unlikely to send back a correct response, in which case, the essage is classified as spa. Our siulation results show the trade-off between the accuracy of anti-spa classifiers and the incurring traffic overhead, and deonstrate that our hybrid fraework is capable of achieving high accuracy regardless of the content-based filtering algorith being used. ª 29 Elsevier Ltd. All rights reserved.. Introduction Short Message Service (SMS) and Multiedia Messaging Service (MMS) are a popular eans of obile counication. Texting costs have decreased continuously over the years (to an extent of free texting) whereas the bandwidth for counication has increased draatically. Such trends have attracted a large nuber of phishing and spaing attacks using SMS essages. In particular, spa containing pornographic or prootive aterials are an eerging phenoenon and they have caused a significant level of inconvenience for users. These are now prevalent in Korea, Japan and China and prone to spread across countries where obile counication is popular. Statistics for 28 (He et al., 28) show that a user in China, on average, receives 8.29 SMS spa per week. Much of the existing research into anti-spa solutions, however, has focused on spa eails. Soe of the popular ethods include white and black listing, digital signature, postage control, address anageent, collaborative and content-based filtering (Healy et al., 25; Metsis et al., 26; Bratko et al., 26; Corack et al., 27; Dwork et al., 23; Hall, 998; Golbeck and Hendler, 24; Androutsopoulos et al., 2). Different characteristics between eails and SMS essages ake it harder for one to apply such approaches directly in obile networks and analyze the results (Deng and Peng, 26). For exaple, the extra traffic required to perfor challenge-response needs to be iniized (or needs to be copensated for) as it is ore expensive to use the bandwidth in obile networks. Also, applying content-based filtering ethods to SMS essages is a challenging task since a obile text essage d containing only a sall text and phone nuber d is relatively shorter in length and contains less structured fields copared to an eail. With eails, additional fields like attachents, links, and iages are coonly used for detecting spa. However, these are not available in SMS essages to construct filtering rules that are as effective as ones used for eails. Due to various drawbacks associated with challenge-response and content-based * Corresponding author. E-ail addresses: yoonj@tcd.ie (J.W. Yoon), hk33@cl.ca.ac.uk (H. Ki), jun.ho.huh@colab.ox.ac.uk (J.H. Huh) /$ see front atter ª 29 Elsevier Ltd. All rights reserved. doi:.6/j.cose.29..3

2 coputers & security 29 (2) filtering, it would ake ore sense to use a cobination rather than just relying on one. In this paper, we propose a spa filtering fraework based on cobination of these two ethods and deonstrate that our cobined approach can be ore effective and efficient in handling spa SMS essages. Using the content-based filtering approach, obvious spa are filtered first to reduce the nuber of essages subject to challenge-response; the challenge-response protocol then classifies achine-generated spa with high accuracy. By cobining the content filtering algorith with the challenge-response schee, we show that, ultiately, high accuracy and low essage traffic can be achieved siultaneously. We also describe four challengeresponse protocols based on Copletely Autoatic Public Test to tell Coputer and Huans Apart (CAPTCHA). Even though any researchers have discussed CAPTCHA based challenge-response protocols (Roan et al., 26; Shirali- Shahreza and Movaghar, 28; He et al., 28), their protocols do not consider the cryptographic details. We extend these protocols for foral verification under a security threat odel. Moreover, our siulation results (see Section 4) show that this hybrid approach is capable of controlling highvolue spa and traffic usage. The reainder of the paper is organized as follows. Section 2 discusses the related work. Section 3 describes the hybrid filtering fraework. Section 4 evaluates the perforance of the proposed fraework based on two easures: traffic usage and accuracy. Finally, Section 5 discusses the contribution of this paper and entails the reaining work. happen ore frequently with SMS essages than with eails due to their saller size and sipler content. Moreover, adaptive schees as such are fundaentally weak against innovative attacks where strategies constantly evolve to anipulate classification rules. Filtering alone will not be sufficient to detect spa. Many anti-spa solutions (He et al., 28; Shirali-Shahreza and Movaghar, 28) have been suggested based on a challenge-response protocol. A essage sender needs to prove that they are a huan user sender by answering the challenge essage (e.g. through a web interface) before their essage is forwarded to the recipient. The senders authenticate theselves as a huan user by answering a siple Turing test which a achine cannot easily understand. The protocol, however, has often been criticized for extra user interaction and traffic used. There ight also be a significant overhead in storing and anaging challenge essages. Roan et al. (26) have introduced a pre-challenge ethod to overcoe these probles. Their ethod assues that each user has a challenge associated with their eail address. Hence, the eail sender can instantly access the recipient s challenge, and send the response together with the eail. Their security odel is underined, however, when the response is exposed to an adversary. He et al. (28) proposed a fraework which cobines white/black listing and challenge-response ethods. However, their work does not consider the necessary security, perforance, and cost iplications of using such a protocol in detail. 2. Related work Content-based filtering solutions have been proved to be effective against eails (Androutsopoulos et al., 2; Metsis et al., 26; Bratko et al., 26), which are typically larger in size copared to SMS essages. Abbreviations and acronys are used ore frequently in SMS essages and they increase the level of abiguity. This akes it difficult to adopt traditional spa filters without any odification. Healy et al. (25) discuss the probles of perforing spa classification on short essages by coparing the perforance of the well-known K-Nearest-Neighbor (KNN), Support Vector Machines (SVM), and Naive Bayes classifiers. They conclude that, for short essages, the SVM and Naive Bayes classifiers substantially outperfor the KNN classifier; and this contrasts with their previous results obtained for longer eails. Hidalgo et al. (26) also carried out content filtering experients with English and Spanish spa SMS corpora to prove that Bayesian filtering ethods are still effective against spa SMS essages. Deng and Peng (26) designed a distributed, content-based filtering ethod that considers other SMS essage characteristics such as its length, which is usually longer than that of a ha (noral essage). One of the drawbacks of existing solutions, however, is that they often look for topical ters or phrases such as free or viagra to identify spa essages. In consequence, soe of the legitiate SMS essages that contain such black listed words can be istakenly classified as spa. This could 3. A hybrid fraework This section describes our hybrid approach. SMS essages are first classified into three different regions using the contentbased filtering ethod: ha, uncertain and spa. Considering that the filtering ethod is not suitable for dealing with uncertain essages, the challenge-response ethod is then used to further classify the uncertain essages into ha and spa regions. In practice, the ajority of spa essages are generated by achines. Therefore, a huan verification echanis d in the for of challenge-response d is used to detect whether an uncertain essage falls into the ha or spa region. Fig. shows a high level overview of three ajor stakeholders: the essage sender, essage center, and recipient. The essage center (owned by the obile operator) sends a challenge query to check whether the sender is a huan or achine. The sender responds by answering the query and the essage center copares the returned value against the known correct value. If the values atch, the essage is classified as ha, otherwise, it is classified as spa. We are interested in further classification of this uncertain region. We would suggest that the essage center should be given the full responsibility of running our fraework for the following reasons: to reduce the traffic usage by filtering spa essages at the earliest possible stage; that is, before forwarding the to the recipient.

3 448 coputers & security 29 (2) Fig. Hybrid spa filtering overview. by using the challenge-response protocol, the essage center will be able to collect a large aount of saple data in real tie; these can be used to develop highly effective classifiers and continuously iprove the perforance of filtering algoriths. it would be difficult to install and aintain a hoogeneous anti-spa software on all obile devices; instead we rely on one solution deployed in the essage center. In practice, however, it is possible that the operator of the essage center would allow certain copanies to send spa essages to its users for a payent. Our work assues that the operator always works in the best interest of the user and will only allow such essages to go through if the user has agreed to receive essages fro these copanies. We iagine that the essage center holds the user s white list of interesting copanies and only forwards essages fro the listed copanies. 3.. Introducing the uncertain region If we assue there are only two regions d ha and spa d the content-based filter will use binary classification. Suppose that we have a probabilistic odel for the anti-spa classifier as a posterior distribution Prðc ¼ hajyþ. This is the probability that a essage falls into the ha region: c and y denote realization of rando variables for a class and essage, respectively. The odd ratio of the posterior is used to obtain a easurable classification by O post ¼ Prðc ¼ hajyþ= Prðc ¼ spajyþ. If O post >, a essage is classified as ha, otherwise, as spa. Alternatively, we can siply use a threshold based approach in the posterior distribution. If Pr(c ¼ hajy) is closer to one, a essage is likely to be ha; if closer to zero, it is likely to be spa. Let c ¼ fðy; hþ be the content-based filter where c and h are the output and given threshold, respectively. This filter would work with the following rules: ha if Prðc ¼ hajyþ h c ¼ fðy; hþ ¼ () spa if Prðc ¼ hajyþ < h This separates ha fro spa (the odd ratio approach is a special case where h ¼.5). The ain proble with this approach is finding a proper threshold: because the threshold for ground truth ~ h is unknown, there are two possible cases as shown in Fig. 2(a). If h is higher than ~ h, soe of the ha in region A could be classified as spa. If h is lower than ~ h, soe of the spa in region B could bypass the content-based filter and reach the recipients. Such a threshold proble will always be present in classification: it is alost ipossible to find the underlying ~ h, and the anti-spa software copanies are likely to use strategies based on their own experiences. In order to iniise the false negatives (i.e. ha being classified as spa and not reaching the recipient) in obile networks, binary ethods tend to be configured with less sensitivity with regards to detecting spa. They would rather istakenly forward spa than prevent any legitiate essage fro reaching the recipient. We believe that these probles can be resolved by introducing an uncertain region with two thresholds (see Fig. 2b). These can be ipleented as the upper and lower boundaries of a traditional threshold syste. As a result, we now have three labels: spa, uncertain area, and ha d the focus is on the uncertain area. Spa and ha regions are classified as in the traditional syste. Only the essages that fall into the uncertain area are checked further using the challenge-response protocol. The next section describes our proposed protocols in detail Challenge-response protocols First, we assue that there is a Turing test available with a low probability of producing false positives and false negatives. CAPTCHA is a coonly used one d it generates pattern atching probles for which a huan can easily recognize and solve, whereas a achine cannot. An autoated progra that generates thousands of spa will not be capable of answering a CAPTCHA based challenge, which could be a graphical iage containing a faint typeface. If the response is correct, there is a high probability that the sender is a huan. CAPTCHA can be designed in different edia

4 coputers & security 29 (2) a Pr(c=ha y) b Pr(c=ha y) A spa noral 2 B spa uncertain area ha anaging session inforation between all trusted pairs for challenge-response would ipose huge storage overhead on the essage center; there ight be ore than one essage center sharing this inforation, and it ight or ight not be stored in the center. Mindful of these security and scalability issues, we proposed four different protocols: protocols 3 and 4 assue an authenticated channel, whereas protocols and 2 do not; protocols and 3 assue that the essage center anages the session inforation, whereas the others do not Protocol notations Standard engineering notations (Burrows et al., 989) were used in describing the protocols. In a protocol that is used by A and B, A/B : X iplies that A sends essage X to B. The sybols S and R represent the Sender and Recipient, respectively. M represents the Message center, T a Tiestap, N a Nonce, K a Key and K its inverse. In a syetric cryptosyste such as AES, K and K are always equal. A Plain SMS essage encrypted with K is represented as {P} K. H is a oneway hash function. The subscript in K iplies that K is M s public key. Additionally, s in K s shows that K s is intended for counication between M and S. The sender s ability to send a correct response depends on their copetence to interpret the key, K c. An unauthorized sender (e.g. a progra sending spa) will not be able to interpret and figure out K c d this key serves to identify achine-generated spa. For siplicity, encryption algoriths were not considered in the protocols. Fig. 2 (a) Two possible cases: h> ~ h (case ) and h< ~ h (case 2) for a given ground truth ~ h (red dot line) and (b) odified classification ebedding uncertain area given a ground truth ~ h (red dot line). fors such as an iage, an audio file or a text (von Ahn et al., 28). Their ipleentation details, however, are beyond the scope of this paper. A nuber of challenge-response protocols have already been proposed (He et al., 28; Shirali-Shahreza and Movaghar, 28). Although, these focus only on the ipleentation issues without considering the security odel and cryptographic details. We define our own security odels and describe a nuber of possible protocols in line with the. There are several issues we need to consider before designing the protocols: when we are dealing with spa, essage authentication and integrity are iportant, whereas confidentiality is not. SMS essages are usually unencrypted and unsigned; hence, it is possible to taper with the during transission. security properties of the counication channel between the essage center and the sender need to be defined; this channel ight or ight not be an authenticated one. The adversary s goal is to deliver spa essages to the recipient Protocols In protocol, the essage center (M ) aintains the session inforation. [Protocol ] (M) S/M : S; R; P (M2) M/S : M; S; fk s g Kc ; fhðs; R; PÞ; Ng Ks (M3) S/M : S; M; fhðs; R; PÞ; N þ g Ks Before sending essage, S stores R and P to prevent essage odification attacks. After receiving essage, M generates K s and stores (S, R, P, K s, N ) as the session inforation. K s is protected with K c. An iage CAPTCHA would be one way of protecting K s against spa progras. After receiving essage 2, S decrypts fk s g Kc by answering the challenge (their ability to interpret K c ). S then decrypts H(S, R, P) and N using K s. S copares H(S, R, P) against the previously stored values. S terinates the protocol if these values do not atch; otherwise, S generates fhðs; R; PÞ; N þ g Ks by K s and sends it to M. After receiving essage 3, M verifies fhðs; R; PÞ; N þ g Ks.Ifitisvalid,M forwards the stored essage (S, R, P) tor. Finally, M deletes the session inforation. The proof of this protocol is presented in Appendix A. The users could becoe frustrated, however, if they receive too any challenge essages. We use a tiestap (T ) to solve this proble. After receiving essage 3, M aintains a session inforation (S, R, P, K s, T ) between S and R for a given tie interval. M checks the validity of K s using the session inforation and a policy that defines the lifetie

5 45 coputers & security 29 (2) of K s. This is also effective for detecting and avoiding any replay attacks. The ain drawback of this protocol is that M has to bear the huge overhead of aintaining the session inforation. We describe another protocol which solves this issue by using authorized tokens instead: [Protocol 2] (M) S/M : S; R; P (M2) M/S : M; S; fk s g Kc ; fhðs; R; PÞg Ks ; fk s ; HðS; RÞ; Tg K (M3) S/M : S; R; fpg Ks ; fk s ; HðS; RÞ; Tg K The ain difference is the use of fk s ; HðS; RÞ; Tg (which K can only be generated by M ) as the authorization token for verifying a response. M checks whether S is authorized by looking at fk s ; HðS; RÞ; Tg K. Using this token, S can just send essage 3 alone, including a new text (P ), within the lifetie of T: (M) S/M : S; R; fp g Ks ; fk s ; HðS; RÞ; Tg K In these protocols, however, S cannot verify the authenticity of the challenge essage. Before describing the next two protocols which ai to solve this proble, we ake an assuption that there is an authenticated channel between M to S, and M s public key (K ) is securely installed on a obile device owned by S (perhaps during the process of anufacturing). We describe the following protocols based on this assuption: [Protocol 3] (M) S/M : S; R; P (M2) M/S : M; S; ffk s g Kc ; Ng K (M3) S/M : S; R; fn þ g Ks In protocol 3, M aintains the session inforation, (S, R, P, K s, N ). When essage 2 arrives, S verifies the signature on ffk s g Kc ; Ng K. S does not respond if the signature is invalid. [Protocol 4] (M) S/M : S; R; P (M2) M/S : M; S; ffk s g Kc ; HðS; RÞ; Tg ; fpg K K (M3) S/M : S; R; fpg Ks ; ffk s g KC ; HðS; RÞ; Tg K Protocol 4 uses ffk s g KC ; HðS; RÞ; Tg K as the authorized token. Our protocols are likely to be copatible with existing devices since the ajority already have built-in encryption and hash functions Observations Upgrading protocols A essage is always sent to the essage center of the contracted operator first. If the essage is directed at soeone contracted to a different operator, it is forwarded to another essage center before reaching the recipient s handset (Enck et al., 25). This eans if one of the essage centers decides not to use our fraework, all uncertain texts delivered via that center would bypass the content-based filter. It would be the weakest point (and the only route needed) for an attack. Hence, all existing essage centers would have to support the new protocol. While this is a large change and a challenging one, operator-sponsored forus like OMTP (Open Mobile Terinal Platfor), are working with key obile operators to unify and recoend obile terinal requireents (Rogers, 27). With the increasing nuber of spa texts, it sees likely that the ability to filter achine-generated uncertain texts will persuade operators into upgrading their systes Perforance degradation If there are too any essages subject to challenge-response, its overhead will doinate. For exaple, sending an iage CAPTCHA is a huge overhead to authenticate a character SMS essage. Future work ay look at adding a bypass to the hybrid, so that a essage originating fro a verified sender can be autoatically treated as ha without having to go through the spa filtering process. For instance, the essage center could anage the recipient s white list of acceptable phone nubers d typically, through synchronization with the recipient s contact list. Since the essage center has secure access to the essage sender details (including the phone nuber), it can first check to see if the sender s phone nuber is included in the recipient s white list. If it is a listed nuber, the essage can be treated as ha and forwarded to the recipient; if not, the essage can go through the spa filtering process. As the uncertain region becoes saller, we expect the perforance of our fraework to iprove Usability issues Adapting CAPTCHA ethods will have iplications on usability. A obile device ight not have the capability to display an iage CAPTCHA to a readable standard; also a obile user ight find it difficult to verify an audio CAPTCHA due to background noise d hence, it is iportant to set up user-friendly CAPTCHA ethods. Different approaches for generating user-friendly CAPTCHA essages have been discussed by various researchers (Leveraging the CAPTCHA Proble, 25; Yan and El Ahad, 28). Chow et al. (28) proposed a new CAPTCHA technique that iniizes the level of user frustration and facilitate the use of CAPTCHA on obile devices. Their technique is well suited for keyboard-less obile devices. 4. Evaluation Fig. 3 deonstrates a basic SMS deployent architecture and its wireless network coponents (Prieto et al., 24): the Hoe Location Register (HLR), Mobile Switching Center (MSC), SMS Gateway (SMSG), and SMS Center (SMSC) are such coponents. These are interconnected as shown in Fig Description of datasets In order to easure the perforance of our fraework, we generated synthetic datasets. Suppose that there are N sent essages (we set N ¼ 5). We use p and q to show the ha to spa proportion where p þ q ¼, and p and q are non-negative nubers (in reality, different operators will have different

6 coputers & security 29 (2) Fig. 3 SMS deployent architecture. proportions). Let k be a rando variable generated fro an existing filtering ethod, given an observed data y: k ¼ Prðc ¼ hajyþ. For an artificial dataset, we build a ixture odel given by pðkjlþ ¼pðkjc ¼ ha; lþpðc ¼ hajlþ þ pðkjc ¼ spa; lþpðc ¼ spajlþ (2) where l denotes a set of hyper-paraeters which control paraeters. Since c can only be (spa) or (ha), we assue c i is generated fro Bernoulli distribution with hyperparaeters p and q. Thus, we have: cwpðcjlþ ¼Bernoulliðc; pþ ¼p c ð pþ c ¼ p c q c After classifying the ith saple essage, we generate the expected probability (this is the filtering output): pðkjc ¼ ha; lþ ¼Bðk; a ; b kwpðkjc; lþ ¼ Þ pðkjc ¼ spa; lþ ¼Bðk; a ; b Þ Beta distribution ðbþ was used here: k denotes the probability of ha classified fro the existing filtering ethod, so the rando variable lies between and ; k can be designed by beta distribution to continuously generate saples between and. Both thresholds (h and h 2 ) vary between and by / 3. In practice, the hyper-paraeters, a, b, a and b, are obtained by the eans and variances of spa and ha respectively; this is given by: a i ¼ i þ 2 i ð iþ s 2 i b i ¼ a i i where i and s i are eans and standard deviation of the costs/ likelihood of spas (i ¼ ) and has (i ¼ ). (3) For an exaple of this paper, we use ¼.375, s ¼.743, ¼.64 and s ¼.597 and then the hyperparaeters of beta distribution are set as follows: a ¼ 3, b ¼ 5, a ¼ 5, b ¼ 2. Also, we built an artificial dataset based on a Spanish database (Hidalgo et al., 26) which shows the proportion of spa as 4.57% and ha as 85.32% d that is, q ¼.457 and p ¼ Fig. 4 shows the distribution of generated data: in graph (a), the red crosses represent ha and blue circles represent spa; the sae colouring schee is used in graph (b). These graphs show that there is a large aount of overlapping labels between.2 and.8. This overlapping section is considered as the uncertain region. Since the challenge-response protocol is not perfect, soe spa will bypass the protocol with correct responses, and soe ha will be filtered istakenly with incorrect responses. To odel this iperfection, we use e and e 2 to represent the ratios of False Positives (FP) and False Negatives (FN) in the uncertain region. In addition, Sections 4.4 and 4.5 use a wide range of randoly generated paraeters to deonstrate how perforance is affected in various environents: Section 4.4 studies perforance with varying ha and spa proportions; Section 4.5 uses fixed proportions of ha and spa (q ¼.457 and p ¼.8532), and varies other paraeters to study how perforance changes Traffic usage coparison We siulated the traffic usage using the variable thresholds and analyzed the results. Our fraework considers several stakeholders (see Fig. 3): the essage Sender (S), essage Receiver (R), essage center (either MSC or SMSG), and other network coponents (SMSC, HLR). First, we calculated the traffic used by an existing filtering ethod. In practice, the size of each essage in the protocol will be different. For instance, the size of the challenge

7 452 coputers & security 29 (2) a Pr(c=Ha y) b The nuber of essages Has Spas Has Spas Index nuber, i for y i essages are deleted at the essage center. Suppose that y h for type {ha, spa} denotes all essages filtered as c¼type type in ters of h, then the total aount of traffic used is the su of jy h c¼ha j6; and jyh j where j$j represents the c¼spa cardinality of a set: N FilteringOnly ¼jy h c¼ha j6 þjyh c¼spa j. This is because a ha traverses through six different paths (S / MSC/SMSG / SMSC / HLR / SMSC / MSC / R) whereas a spa just traverses through one (S / MSC/SMSG). In contrast, our hybrid odel divides the easurable space into three different areas using two thresholds: h and h 2.As a result, we have two ore paraeters to estiate: the traffic used by ha (N un ) and spa (N us ) in the uncertain region. Let y ~c¼type for type {ha, spa} be a set of essages that have label type as a ground truth. As Fig. 5 shows, there are four possible pathways: in (a), a essage classified as ha (using the higher threshold) is sent directly to R via the network coponents; the nuber of paths taken is six: S / MSC/SMSG / SMSC / HLR / SMSC / MSC / R. in (b), a essage is in between the higher and the lower thresholds; a correct response is subitted by the sender and the essage is classified as ha; the nuber of paths taken is eight: S / MSC/SMSG / S / MSC/SMSG / SMSC / HLR / SMSC / MSC / R. in (c), a essage is in between two thresholds again; this tie no response is returned and the essage is classified as spa; the nuber of paths taken is two: S / MSC/SMSG / S. in (d), a essage, classified as spa using the lower threshold, is deleted at the essage center (MSC/SMSG); the nuber of paths taken is one: S / MSC/SMSG Pr(c=Haly) Fig. 4 Displaying k [ Pr(c [ hajy) for N essages: spa (4.57%) and ha (85.32%). (a) N essages and (b) distribution. essage will be greater than other text essages since it would include a CAPTCHA iage. For siplicity, however, we assue that all essages have the sae size. Only the essages with filtering probabilities higher than the threshold h reach R via the network coponents; other The traffic usage is calculated using: N n ¼ jy h 2 j 6 c¼ha N un ¼ y h þ yh N us ¼ yh þ c¼ha Xyh 2 Xy c¼spa ~c¼ha c¼ha Xyh 2 Xy c¼spa ~c¼spa c¼ha Xyh 2 Xy c¼spa ~c¼spa y h c¼ha Xyh 2 Xy c¼spa ~c¼ha N s ¼ jy h c¼spa j N hybrid ¼ N n þ N un þ N us þ N s ð e Þ8 e 2 8 ð e 2Þ 2 e 2 where e is a probability that huans fail to respond correctly and e 2 is a probability that spa generated by achines pass (4) Fig. 5 Four possible pathways for the hybrid ethod.

8 coputers & security 29 (2) a Turing test. Again, for siplicity, we assue that these probabilities are relatively low and set e and e 2 at.2 and., respectively. Fig. 6 shows the traffic usage and accuracy with varying thresholds, h and h 2. We assue that h is saller than h 2, and only the right half of the graph is eaningful since the right and left halves of the graph are syetric. The green plane represents the traffic used by the filtering ethod alone, and the blue represents the traffic used by our hybrid fraework. In order to show the changes in traffic usage with two varying thresholds, the inner sections of Fig. 6 were explored further in Fig. 7. Graph (a) was plotted with the higher threshold fixed to.73333, and with the lower threshold increasing fro until it reached this value. The graph shows that the traffic usage decreases as the lower threshold increases. Additionally, the traffic usage ratios of our hybrid approach and the conventional approach (filtering only) are :37 ð¼ 4:=3Þ and :27 ð¼ 3:3=2:6Þ respectively (at low thresholds and.5). Fro this, we concluded that the aount of traffic used in our approach is roughly.3 ties greater than that of the conventional approach. We also onitored the traffic usage with the lower threshold fixed to ¼., and with the higher threshold increasing fro. to (see graph (b) in Fig. 7). The traffic usage does not change with the filtering-only approach because the lower threshold is the sae as h. As the nuber of essages in the uncertain region increases so does the traffic usage ROC coparison One of the good easures used in classification is Receiver Operating Characteristic (ROC) curves. We calculated and copared True Positive (TP), True Negative (TN ), False Positive (FP) and False Negative (FN ) of the underlying classes and the expected ones between the filtering-only ethod and our hybrid ethod. Let q be {TP, TN, FP, FN}. We obtained the proper estiate of q fro the posterior distributions: p(qjh) was used for the filtering ethod and p(qjh, h 2 ) for the hybrid ethod. q was obtained in the filtering-only ethod by a x b ACC h 2.4 h2.2.2 ~ qfiltering only ¼ Eðqjh; yþ (5) where Eð$jh; yþ denotes expectation given a threshold. We also obtained q in the hybrid ethod by Eðqjh ; h 2 ; yþ, which denotes expectation given two thresholds. We used arginalized h.4 h N total =N un +N us +N s N Filtering Only ACC hybrid ACC Filtering Only Fig. 6 3D view of the (a) traffic usages and (b) accuracy in ters of varying thresholds..8 a 6 x 4 A fixed high threshold: b 6 x 4 A fixed low threshold:. The total traffic path length The total traffic path length Low thresholds High thresholds Ha, N n Ha in uncertain, N un Spa in uncertain, N us Spa, N s N hybrid =N un +N us +N s N Filtering only Ha, N n Ha in uncertain, N un Spa in uncertain, N us Spa, N s N total =N un +N us +N s N Filtering only Fig. 7 Slides of an axis (with fixed threshold). (a) A fixed high threshold and (b) a fixed low threshold.

9 454 coputers & security 29 (2) posterior distribution for the hybrid ethod since the nuber of thresholds need to be equal for a fair coparison. ~ qhybrid ¼ Eðqjh ; yþ ¼ R qpðqjh ; yþdq q ¼ R h R i q pðq; h h2 2jh ; yþdh 2 dq q ¼ R " # R q pðqjh 2 ; h ; yþpðh 2 jh Þdh 2 dq q h 2 ¼ R R qpðqjh 2 ; h ; yþpðh 2 jh Þdh 2 dq q h 2 ¼ R R qpðqjh 2 ; h ; yþdq pðh 2 jh Þdh 2 h 2 q P R z jh 2 qpðqjh j 2 ; h ; yþdq h 2 H 2 q P ¼ jh 2j Eðqjh ; h 2 ; yþ h 2 H 2 h 2 w p(h 2 jh ) and H 2 are a set of saples h 2. Fro Eq. (5) and Eq. (6), we plotted an ROC curve with the threshold increasing fro to (see Fig. 8); x-and y-axis stand for -specificity and sensitivity; these are estiated by specificity ¼ TN and sensitivity ¼ TP FP þ TN TP þ FN : (7) The plain black line is used to show the filtering-only ethod. The coloured lines with arkers are used for the others that use the hybrid ethod. We have tested four different e s and e 2 s: e {.2,.4} and e 2 {.8,.}; these are shown with the coloured lines. This graph shows that our hybrid ethod has higher perforance than the other. The ROC can also be used to generate a suary statistic. One of the coon versions is the Area Under the ROC Curve (AUC). The AUC corresponds to the probability of a classifier ranking a randoly chosen positive instance higher than a negative one. The coparison of AUC for all ethods is described in Table. In this table, the AUCs of all hybrid ethods are higher than that of the filtering ethod. This ephasizes our previous result of the hybrid ethod having a superior perforance. In addition, as the ratios of e and e 2 becoe saller, the AUC increases. (6) Table Coparison of AUC. Method Ratio (e ) Ratio (e 2 ) AUC Filtering only.926 Hybrid Hybrid Hybrid Hybrid Variant proportion of spa Previously, in Sections 4.2 and 4.3, the proportions of spa and ha were fixed to 4.57% and 85.32% respectively. In this section we show how the perforance is affected when these proportions change. Table 2 describes a sall nuber of saples fro the nine different proportions. Each record has six different coluns: proportion of spa (%), lower threshold (h ), higher threshold (h 2 ), traffic usage (TU) of N hybrid, ratio ð¼ N hybrid =N filteringonly Þ, and accuracy ðacc ¼ðTP þ TNÞ=ðP þ NÞÞ. It uses three different easures for the perforance. If the traffic usage is less, we say the syste is lighter and is ore econoical. The ratio is only close to if the traffic used in the hybrid ethod is close to the aount used in the other. The accuracy easures the correctness of essage classification. We can select practical threshold values for each spa proportion to copare the perforance. For instance, threshold values h ¼. and h 2 ¼.2 can be selected in % spa proportion to show a reasonable perforance of the hybrid ethod. However, if the syste is concerned with achieving high accuracy and not with reducing the traffic usage, h ¼. and h 2 ¼.9 values can be used. In a spa-doinant environent (for spa Table 2 Traffic aounts and accuracy of hybrid ethods in ters of thresholds. Proportion of spa h h 2 TU Ratio ACC Sensitivity Filtering Only, [AUC:.9268].2 FP(uncertain):.2, FN(uncertain):.8[AUC:.9783] FP(uncertain):.4, FN(uncertain):.8[AUC:.97362]. FP(uncertain):.2, FN(uncertain):.[AUC:.9788] FP(uncertain):.4, FN(uncertain):.[AUC:.97349] Specificity Fig. 8 ROC curve. %..2 3, , , , %..2 3, , , , %..2 3, , , , %..2 3, , , , %..2 3, , , ,

10 coputers & security 29 (2) proportion of 5%), reasonable threshold values would be h ¼.4 and h 2 ¼.6. Returning back to the figures for a spa proportion of %, h ¼. and h 2 ¼.9 will be selected when accuracy is the ost significant factor Different content-based filtering paraeters and perforance iplications The perforance (accuracy and traffic usage) of the proposed hybrid ethod will vary depending on the characteristics of the content-based filtering ethod being used. In order to study this further, we siulated the hybrid ethod with 2 rando saples, each representing a different case of the content-based filtering ethod. The following paraeters were considered for each saple: : the ean of the cost/likelihood of spa s : the standard deviation of the cost/likelihood of spa : the ean of the cost/likelihood of ha s : the standard deviation of the cost/likelihood of ha h : a lower threshold h 2 : a higher threshold. Fro the results, their accuracy and ratio of traffic usage (hybrid ethod to conventional filtering-only ethod) values were plotted in a graph (see Fig. 9). The results were clustered using a well-known clustering technique called the K-eans algorith (Hartigan, 975) by setting K ¼ 4 (iplying 4 clusters). The saples clustered around the top left region of the graph (assigned with label ) are regarded as recoendable cases since they show high accuracy but with saller increase in the traffic usage copared to the filtering-only ethod. The saples clustered around the top right region of the graph (assigned with label 2) show high accuracy but suffer fro a large increase in the traffic usage copared to the filtering-only ethod. This iplies that the hybrid ethod should be used for these saples when ACC Ratio of traffic Fig. 9 Accuracy and traffic usage ratio for 2 rando saples d each assigned with one of the four labels: label (red dots, top left), label 2 (black crosses, top right), label 3 (blue squares, botto left) and label 4 (pink triangles, center). achieving a high accuracy is considered relatively ore iportant than the resulting increase in the traffic usage. The worst set of saples are those clustered around the botto left region of the graph (assigned with label 3). Although these show sall increase in the traffic usage, the accuracy is also very low. For this reason, the hybrid ethod is not really suitable for handling such saples. The saples clustered broadly around the center region of the graph (assigned with label 4) are considered better than those assigned with label 3, but worse than those assigned with labels and 2. More can be observed fro an associated set of data, presented in Table 3 (see Appendix B). Each record describes a saple profile. 2 Different applications and business odels will have different preferences for these two ratios. Taking this into consideration, we studied the trends for three representative cases where. accuracy is considered twice as iportant as traffic usage Fig. (a) 2. both are considered as equally iportant Fig. (b) 3. traffic usage is considered twice as iportant as accuracy Fig. (c) The results are shown in Fig.. In Fig. (a), the saples which satisfy D ACC >.5D TU þ.5 are plotted with red dots and these represent good saples. The others, plotted with black x s, are classified as relatively bad saples. Siilarly, in Fig. (b) and (c), good saples (red dots) satisfy D ACC > D TU and D ACC > 2D TU, respectively, whereas relatively bad ones (black x s) do not. We also plotted the ratio between the nuber of good saples and bad saples as derived fro these three cases (see Fig. ). The graph shows that, with the varying iportance of these two factors, the ratio of the nuber of good saples to bad saples changes: as the iportance of accuracy (relative to traffic usage) increases, so does the nuber of good saples, and vice versa. Figs. and both show that as the traffic usage becoes ore iportant, the nuber of good saples decreases. Hence, if our hybrid ethod was to be applied in an environent where traffic usage is considered relatively ore iportant, the content-based filtering paraeters as well as the threshold values need to be selected ore carefully. 2 A saple profile consists of the specific paraeter values. Given the space available, we only show representative saples for each label (L). Consider the saples that have been assigned with labels 3 (L ¼ 3) and 4 (L ¼ 4). In practice, there is a very low chance for these cases to arise since the eans of spa and ha ( and ) are lower than the lower threshold, and the standard deviations are also relatively sall. Therefore, if we study the graph without being too concerned about such cases, it becoes clearer that our hybrid ethod, in general, is capable of achieving high accuracy regardless of the content-based filtering algorith (or the paraeters) being used. We also studied the configurations of the hybrid ethod with respect to the ratio of accuracy ðd ACC ¼ ACC hybrid =ACC filteringonly Þ and the ratio of traffic usage ðd TU ¼ TU hybrid =TU filteringonly Þ.

11 456 coputers & security 29 (2) a Ratio of the ACC.5..5 b Ratio of the ACC Ratio of the traffic c Ratio of the traffic.5 Ratio of the ACC Ratio of the traffic Fig. Coparison between the accuracy ratio and the traffic usage ratio for 2 saples d dotted blue lines stand for the borderline between the good saples and bad saples. (a) D ACC :D TU [ :2. (b) D ACC :D TU [ :. (c) D ACC :D TU [ 2:. 5. Conclusion and future work We proposed a hybrid spa filtering fraework for obile counication using a cobination of content-based filtering and challenge-response. A essage that falls into the uncertain region (after filtering), is further classified by sending a challenge (e.g. an iage CAPTCHA) to the sender: a legitiate sender is likely to answer it correctly, whereas an autoated spa progra is not. Challenge-response protocols have been carefully designed with the necessary cryptographic features. We have also shown the trade-off between accuracy and traffic # of good saples/ # of bad saples Δ ACC /Δ TU Fig. The ratio between the nuber of good saples and bad saples. usage in using our fraework and copared it with the conventional content-based filtering ethod. Moreover, through a siulation of 2 randoly generated saples (each representing a unique set of content-based filtering paraeters and threshold values) we showed that our hybrid approach, in general, achieves high accuracy regardless of the content-based filtering algorith being applied. Although, when traffic usage becoes relatively ore iportant than accuracy, the underlying filtering algoriths ust be selected ore carefully. In this paper, a synthetic dataset, as opposed to a real dataset, has been used due to the following two reasons: first, we wanted to consider a wide range of application environents, each of which will require different level of accuracy and traffic usage (e.g. VoIP spa filters Croft and Olivier, 25); and second, this protocol involves a great level of huan interaction and developing such a prototype (in order to generate our own dataset) was outside the scope. As part of the future work, we could contact obile operators and forus like OMTP to collect real data and verify the accuracy of our results. Having the network operators charge for sending of SMS essages has been one of the big inhibitors to the growth of spa: even a cent per essage ight hugely alter the econoics of a spaer. Assuing that a reasonable filtering ethod is in place, another hybrid potential is to force spaers to opt into a charging schee where the cost of responding to a challenge is larger than sending an initial spa. For exaple, if it costs two cents to send a spa, then it would cost extra five cents to answer an iage CAPTCHA. It is difficult to assess how effective a solution this ight be, but future work ay explore these econoic easures in depth as a potential enhanceent to the hybrid approach.

12 coputers & security 29 (2) Acknowledgeents The authors would like to thank the anonyous referees for their careful attention and insightful coents. Appendix A. Proof of protocol Protocol has been proved by BAN logic (refer to the rules given by Burrows et al., 989). In this protocol, M should be able to trust H(S, R, P) returned fro S, and know whether it is a legitiate sender or not. For foral verification, we derive an ideal protocol of protocol. The sybols N s and N represent a sender s nonce and a recipient s nonce, respectively. Our goal is to show that this protocol satisfies the security property (G). [Ideal Protocol ] (M2) M/S : fm 4 Ks Sg Kc ; fn ; ðs; R; PÞg Ks (M3) S/M : fn s ; N ; ðs; R; PÞg Ks [Security Goal] (G) MjhSjhðN s ; N ; ðs; R; PÞÞ Message is ignored since it does not contribute uch to achieving the goal; {N þ } is shown as N s. The initial state assuptions have been derived: (A3) assues that K s will be shared with a legitiate sender capable of interpreting K c. The proof is described as follows: Sending essage 2 leads to: () MjwfM KMS 4 Sg KC (2) MjwfN M ; ðs; R; PÞg KMS (3) S9fM KMS 4 Sg KC (4) S9fN M ; ðs; R; PÞgK MS Sending essage 3 leads to: (5) SjwfN S ; N M ; ðs; R; PÞgK SR (6) M9fN S ; N M ; ðs; R; PÞgK SR (7) is derived fro (A3) and (6) by the essage-eaning rule. (7) MjhSjwðN S ; N M ; ðs; R; PÞÞ (N S, N M,(S, R, P)) contains the nonce, N S, and hence (8). (8) Mjh#ðN S ; N M ; ðs; R; PÞÞ Finally (G) is derived fro (7) and (8) by the nonce-verification rule. In protocol, S cannot verify whether essage 2 is fro their contracted operator. Other protocols can be proved in a siilar anner. (A) Mjh#N (A2) Sjh#N s (A3) MjhM 4 Ks S (A4) SjhMjM 4 Ks S (A5) Mjh / Kc S (A6) Sjh / k c S Appendix B. Traffic aounts and accuracy of hybrid ethods of 2 rando saples (L: labels/classes) Table 3. Table 3 Traffic aounts and accuracy of hybrid ethods of 2 rando saples (L: labels/classes). L ID s s h h 2 TU Ratio ACC eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ

13 458 coputers & security 29 (2) Table 3 (continued) L ID s s h h 2 TU Ratio ACC eþ eþ eþ eþ eþ eþ eþ eþ eþ eþ references Androutsopoulos I, Koutsias J, Chandrinos K, Spyropoulos CD. An experiental coparison of naive Bayesian and keyword-based anti-spa filtering with personal e-ail essages. In: SIGIR : Proceedings of the 23rd annual international ACM SIGIR conference on research and developent in inforation retrieval. New York, NY, USA: ACM; 2. p Bratko A, Filipiè B, Corack GV, Lyna TR, Zupan B. Spa filtering using statistical data copression odels. Journal of Machine Learning Research 26;7: Burrows M, Abadi M, Needha R. A logic of authentication. ACM Operating Systes Review 989;23(5): 3. Chow R, Golle P, Jakobsson M, Wang L, Wang X. Making CAPTCHAs clickable. In: HotMobile 8: Proceedings of the 9th workshop on obile coputing systes and applications. New York, NY, USA: ACM; 28. p Corack GV, Hidalgo JMG, Sanz EP. Spa filtering for short essages. In: Proceedings of the 6th ACM conference on inforation and knowledge anageent; 27. p Croft NJ, Olivier MS. A odel for spa prevention in IP telephony networks using anonyous verifying authorities. In: ISSA, new knowledge today conference; 25. Deng W, Peng H. Research on a naive Bayesian based short essage filtering syste. In: Machine learning and cybernetics, 26 international conference on Aug. 26. p Dwork C, Goldberg A, Naor M. On eory-bound functions for fighting spa. In: Proceedings of the 23rd annual international cryptology conference (CRYPTO 23); August 23. Enck W, Traynor P, McDaniel P, Porta T. Exploiting open functionality in SMS-capable cellular networks. In: CCS, Nov. 25. Golbeck J, Hendler J. Reputation network analysis for eail filtering. In: Proceedings of the conference on eail and antispa (CEAS); 24. Hall RJ. How to avoid unwanted eail. Counications of the ACM March 998. Hartigan JA. Clustering algoriths. New York: John Wiley and Sons; 975. He P, Sun Y, Zheng W, Wen X. Filtering short essage spa of group sending using CAPTCHA. In: Workshop on knowledge discovery and data ining; 28. p Healy M, Delany S, Zaolotskikh A. An assessent of case-based reasoning for short text essage classification. In: Proceedings of 6th Irish conference on artificial intelligence and cognitive science; 25. p Hidalgo JMG, Bringas GC, Sanz EP, Garc FC. Content based SMS spa filtering. In: Proceedings of the 26 ACM syposiu on docuent engineering. Asterda, The Netherlands: ACM Press; October 26. p. 3. Leveraging the CAPTCHA Proble; 25. Metsis Vangelis, Androutsopoulos Ion, Paliouras Georgios. Spa filtering with naive Bayes which naive Bayes? In: Third conference on eail and anti-spa (CEAS); 26. Prieto AG, Cosenza R, Stadler R. Policy-based congestion anageent for an SMS gateway. In: Proceedings of the fifth IEEE international workshop; 24. Rogers D. Mobile handset security: securing open devices and enabling trust. OMTP Liited White Paper 27. Roan R, Zhou J, Lopez J. An anti-spa schee using prechallenges. Coputer Counications 26;29(5): Shirali-Shahreza S, Movaghar A. An anti-sms-spa using CAPTCHA. In: CCCM 8: Proceedings of the 28 ISECS international colloquiu on coputing, counication, control, and anageent. Washington, DC, USA: IEEE Coputer Society; 28. p von Ahn L, Maurer B, Mcillen C, Abraha D, Blu M. RECAPTCHA: Huan-based character recognition via web security easures. Science August 28. Yan J, El Ahad AS. Usability of CAPTCHAs or usability issues in CAPTCHA design. In: SOUPS 8: Proceedings of the 4th syposiu on usable privacy and security. New York, NY, USA: ACM; 28. p Ji Won Yoon. He received the B.Sc. degree in inforation engineering at the SungKyunKwan University, Korea. He obtained the M.Sc. degree in School of inforatics at the University of

14 coputers & security 29 (2) Edinburgh UK in 24 and the Ph.D. degree in signal processing group at the University of Cabridge UK in 28 respectively. In 28, he oved to departent of Engineering science, the University of Oxford, UK to do postdoctoral research. He is currently a Research Fellow with Statistics departent, Trinity College Dublin, Ireland. His research interests include Bayesian statistics, Machine Learning, data ining, Network Security and Bioedical engineering. He has worked on applications in brain signals, cosology, biophysics and ultiedia. Hyoungshick Ki. He received the B.Sc. degree in inforation engineering at the SungKyunKwan University, Korea. He obtained the M.Sc. degree in departent of coputer science, KAIST, Korea in 2. He previously worked for Sasung Electronics as a senior engineer fro May 24 to Septeber 29. He also served a eber of DLNA and Coral standardization for DRM interoperability in hoe networks. He is currently studying in the Coputer Laboratory at the University of Cabridge as a PhD student. His research interest is focused on security and privacy in coplex networks and distributed systes. Jun Ho Huh. He holds Software Engineering and International Business degrees fro Auckland University. He is currently adphilstudentattheoxforduniversitycoputinglaboratory. His research interests include trusted virtualization, trustworthy audit and logging, and security in distributed systes.