Secure routing for structured peer-to-peer overlay networks

Transcription

1 Secure routing for structured peer-to-peer overly networks Miguel Cstro 1, Peter Druschel 2, Aylvdi Gnesh 1, Antony Rowstron 1 nd Dn S. Wllch 2 1 Microsoft Reserch Ltd., 7 J J Thomson Avenue, Cmbridge, CB3 FB, UK {mcstro,jg,ntr}@microsoft.com 2 Rice University, 1 Min Street, MS 132, Houston, TX , USA {druschel,dwllch}@cs.rice.edu Abstrct Structured peer-to-peer overly networks provide substrte for the construction of lrge-scle, decentrlized pplictions, including distributed storge, group communiction, nd content distribution. These overlys re highly resilient; they cn route messges correctly even when lrge frction of the nodes crsh or the network prtitions. But current overlys re not secure; even smll frction of mlicious nodes cn prevent correct messge delivery throughout the overly. This problem is prticulrly serious in open peer-to-peer systems, where mny diverse, utonomous prties without preeisting trust reltionships wish to pool their resources. This pper studies ttcks imed t preventing correct messge delivery in structured peer-to-peer overlys nd presents defenses to these ttcks. We describe nd evlute techniques tht llow nodes to join the overly, to mintin routing stte, nd to forwrd messges securely in the presence of mlicious nodes. 1 Introduction Structured peer-to-peer (p2p) overlys like CAN [1], Chord [2], Pstry [17] nd Tpestry [21] provide self-orgnizing substrte for lrge-scle peer-to-peer pplictions. These systems provide powerful pltform for the construction of vriety of decentrlized services, including network storge, content distribution, nd ppliction-level multicst. Structured overlys llow pplictions to locte ny object in probbilisticlly bounded, smll number of network hops, while requiring per-node routing tbles with only smll number of entries. Moreover, the systems re sclble, fulttolernt nd provide effective lod blncing. However, to fully relize the potentil of the p2p prdigm, such overly networks must be ble to support n open environment where mutully distrusting prties with conflicting interests re llowed to join. Even in closed system of sufficiently lrge scle, it my be unrelistic to ssume tht none of the prticipting nodes hve been compromised by ttckers. Thus, structured overlys must be robust to vriety of security ttcks, including the cse where frction of the prticipting nodes ct mliciously. Such nodes my mis-route, corrupt, or drop messges nd routing informtion. Additionlly, they my ttempt to ssume the identity of other nodes nd corrupt or delete objects they re supposed to store on behlf of the system. In this pper, we consider security issues in structured p2p overly networks. We describe ttcks tht cn be mounted ginst such overlys nd the pplictions they support, nd present the design of secure techniques tht cn thwrt such ttcks. In prticulr, we identify secure routing s key building block tht cn be combined with eisting, ppliction-specific security techniques to construct secure, decentrlized pplictions upon structured overlys. Secure routing requires (1) secure ssignment of node identifiers, (2) secure routing tble mintennce, nd (3) secure messge forwrding. We present techniques for ech of these problems, nd show how using these techniques, secure routing cn be mintined efficiently despite up to 2% of mlicious prticipting nodes. Moreover, we show tht the overhed of secure routing is cceptble nd proportionl to the frction of mlicious nodes. The rest of this pper is orgnized s follows. Section 2 gives some bckground on structured p2p overlys, specifies models nd ssumptions, nd defines secure routing. Sections 3, 4 nd present ttcks on nd solutions for ssignment of identifiers to nodes, routing tble mintennce nd messge forwrding, respectively. Section eplins how the overhed of secure routing cn be minimized by using self-certifying dt. Finlly, Section 7 discusses relted work nd Section 8 provides conclusions. Appers in Proc. of the th Useni Symposium on Operting Systems Design nd Implementtion, Boston, MA, December 22.

2 2 Bckground, models nd solution In this section, we present some bckground on structured p2p overly protocols like CAN, Chord, Tpestry nd Pstry. Spce limittions prevent us from giving detiled overview of ech protocol. Insted, we describe n bstrct model of structured p2p overly networks tht we use to keep the discussion independent of ny prticulr protocol. For concreteness, we lso give n overview of Pstry nd point out relevnt differences with the other protocols. Net, we describe models nd ssumptions used lter in the pper bout how fulty nodes my behve. Finlly, we define secure routing nd outline our solution. Throughout this pper, most of the nlyses nd techniques re presented in terms of our bstrct model, nd should pply to other structured overlys ecept when otherwise noted. However, the security nd performnce of our techniques ws fully evluted only in the contet of Pstry; full evlution of the techniques in other protocols is future work. 2.1 Routing overly model We define n bstrct model of structured p2p routing overly, designed to cpture the key concepts common to overlys like CAN, Chord, Tpestry nd Pstry. In our model, prticipting nodes re ssigned uniform rndom identifiers, nodeids, from lrge id spce (e.g., the set of 128-bit unsigned integers). Applictionspecific objects re ssigned unique identifiers, clled keys, selected from the sme id spce. Ech key is mpped by the overly to unique live node, clled the key s root. The protocol routes messges with given key to its ssocited root. To route messges efficiently, ech node mintins routing tble with nodeids of other nodes nd their ssocited IP ddresses. Moreover, ech node mintins neighbor set, consisting of some number of nodes with nodeids ner the current node in the id spce. Since nodeid ssignment is rndom, ny neighbor set represents rndom smple of ll prticipting nodes. For fult tolernce, ppliction objects re stored t more thn one node in the overly. A replic function mps n object s key to set of replic keys, such tht the set of replic roots ssocited with the replic keys represents rndom smple of prticipting nodes in the overly. Ech replic root stores copy of the object. Net, we discuss eisting structured p2p overly protocols nd how they relte to our bstrct model. 2.2 Pstry Pstry nodeids re ssigned rndomly with uniform distribution from circulr 128-bit id spce. Given 128- bit key, Pstry routes n ssocited messge towrd the live node whose nodeid is numericlly closest to the key. Ech Pstry node keeps trck of its neighbor set nd notifies pplictions of chnges in the set. Node stte: For the purpose of routing, nodeids nd keys re thought of s sequence of digits in bse 2 b (b is configurtion prmeter with typicl vlue 4). A node s routing tble is orgnized into 128/2 b rows nd 2 b columns. The 2 b entries in row r of the routing tble contin the IP ddresses of nodes whose nodeids shre the first r digits with the present node s nodeid; the r + 1th nodeid digit of the node in column c of row r equls c. The column in row r tht corresponds to the vlue of the r + 1th digit of the locl node s nodeid remins empty. A routing tble entry is left empty if no node with the pproprite nodeid prefi is known. Figure 1 depicts n emple routing tble. Ech node lso mintins neighbor set (clled lef set ). The lef set is the set of l nodes with nodeids tht re numericlly closest to the present node s nodeid, with l/2 lrger nd l/2 smller nodeids thn the current node s id. The vlue of l is constnt for ll nodes in the overly, with typicl vlue of pproimtely 8 log 2 bn, where N is the number of epected nodes in the overly. The lef set ensures relible messge delivery nd is used to store replics of ppliction objects. Messge routing: At ech routing step, node seeks to forwrd the messge to node in the routing tble whose nodeid shres with the key prefi tht is t lest one digit (or b bits) longer thn the prefi tht the key shres with the present node s id. If no such node cn be found, the messge is forwrded to node whose nodeid shres prefi with the key s long s the current node, but is numericlly closer to the key thn the present node s id. If no pproprite node eists in either the routing tble or neighbor set, then the current node or its immedite neighbor is the messge s finl destintion. Figure 2 shows the pth of n emple messge. Anlysis shows tht the epected number of routing hops is slightly below log 2 bn, with distribution tht is tight round the men. Moreover, simultion shows tht the routing is highly resilient to crsh filures. To chieve self-orgniztion, Pstry nodes must dynmiclly mintin their node stte, i.e., the routing tble nd neighbor set, in the presence of node rrivls nd node filures. A newly rriving node with the new nodeid X cn initilize its stte by sking ny eisting Pstry node A to route specil messge using X s the key. The messge is routed to the eisting node Z with nodeid numericlly closest to X. X then obtins the neighbor set from Z nd constructs its routing tble by copying rows from the routing tbles of the nodes it encountered on the originl route from A to Z. Finlly, X nnounces its presence to the initil members of its neighbor set, which in turn updte their own neighbor sets nd routing tbles.

3 b b b c c c d d d e e e f f f O d41c d471f1 d47c4 d42b d4213f b c d e f 1fc Route(d41c) d13d3 Figure 1: Routing tble of Pstry node with nodeid 1, b = 4. Digits re in bse 1, represents n rbitrry suffi. Figure 2: Routing messge from node 1 f c with key d41c. The dots depict live nodes in Pstry s circulr nmespce. Similrly, the overly cn dpt to brupt node filure by echnging smll number of messges (O(log 2 bn)) mong smll number of nodes. 2.3 CAN, Chord, Tpestry Net, we briefly describe CAN, Chord nd Tpestry, with n emphsis on the differences reltive to Pstry. Tpestry is very similr to Pstry but differs in its pproch to mpping keys to nodes nd in how it mnges repliction. In Tpestry, neighboring nodes in the nmespce re not wre of ech other. When node s routing tble does not hve n entry for node tht mtches key s nth digit, the messge is forwrded to the node with the net higher vlue in the nth digit, modulo 2 b, found in the routing tble. This procedure, clled surrogte routing, mps keys to unique live node if the node routing tbles re consistent. Tpestry does not hve direct nlog to neighbor set, lthough one cn think of the lowest populted level of the Tpestry routing tble s neighbor set. For fult tolernce, Tpestry s replic function produces set of rndom keys, yielding set of replic roots t rndom points in the id spce. The epected number of routing hops in Tpestry is log 2 bn. Chord uses 1-bit circulr id spce. Unlike Pstry, Chord forwrds messges only in clockwise direction in the circulr id spce. Insted of the prefi-bsed routing tble in Pstry, Chord nodes mintin routing tble consisting of up to 1 pointers to other live nodes (clled finger tble ). The ith entry in the finger tble of node n refers to the live node with the smllest nodeid clockwise from n+2 i 1. The first entry points to n s successor, nd subsequent entries refer to nodes t repetedly doubling distnces from n. Ech node in Chord lso mintins pointers to its predecessor nd to its n successors in the nodeid spce (this successor list represents the neighbor set in our model). Like Pstry, Chord s replic function mps n object s key to the nodeids in the neighbor set of the key s root, i.e., replics re stored in the neighbor set of the key s root for fult tolernce. The epected number of routing hops in Chord is 1 2 log 2N. CAN routes messges in d-dimensionl spce, where ech node mintins routing tble with O(d) entries nd ny node cn be reched in (d/4)(n 1/d ) routing hops on verge. The entries in node s routing tble refer to its neighbors in the d-dimensionl spce. CAN s neighbor tble duls s both the routing tble nd the neighbor set in our model. Like Tpestry, CAN s replic function produces rndom keys for storing replics t diverse loctions. Unlike Pstry, Tpestry nd Chord, CAN s routing tble does not grow with the network size, but the number of routing hops grows fster thn logn in this cse. Tpestry nd Pstry construct their overly in Internet topology-wre mnner to reduce routing delys nd network utiliztion. In these protocols, routing tble entries cn be chosen rbitrrily from n entire segment of the nodeid spce without incresing the epected number of routing hops. The protocols eploit this by initilizing the routing tble to refer to nodes tht re nerby in the network topology nd hve the pproprite nodeid prefi. This gretly fcilittes proimity routing [17]. However, it lso mkes these systems vulnerble to certin ttcks, s shown in Section 4. The choice of entries in CAN s nd Chord s routing tbles is tightly constrined. The CAN routing tble entries refer to specific neighboring nodes in ech dimension, while the Chord finger tble entries refer to specific points in the nodeid spce. This mkes proimity routing hrder but it protects nodes from ttcks tht eploit ttcking nodes proimity to their victims.

4 2.4 System model The system runs on set of N nodes tht form n overly using one of the protocols described in the previous section. We ssume bound f ( f < 1) on the frction of nodes tht my be fulty. Fults re modeled using constrined-collusion Byzntine filure model, i.e., fulty nodes cn behve rbitrrily nd they my not ll necessrily be operting s single conspircy. The set of fulty nodes is prtitioned into independent colitions, which re disjoint sets with size bounded by cn (1/N c f ). When c = f, ll fulty nodes my collude with ech other to cuse the most dmge to the system. We model the cse where nodes re grouped into multiple independent colitions by setting c < f. Members of colition cn work together to corrupt the overly but re unwre of nodes in other colitions. We studied the behvior of the system with c rnging from 1/N to f to model different filure scenrios. We ssume tht every node in the p2p overly hs sttic IP ddress t which it cn be contcted. In this pper, we ignore nodes with dynmiclly ssigned IP ddresses, nd nodes behind network ddress trnsltion boes or firewlls. While p2p overlys cn be etended to ddress these concerns, this pper focuses on more trditionl network hosts. The nodes communicte over norml Internet connections. We distinguish between two types of communiction: network-level, where nodes communicte directly without routing through the overly, nd overly-level, where messges re routed through the overly using one of the protocols discussed in the previous section. We use cryptogrphic techniques to prevent dversries from observing or modifying network-level communiction between correct nodes. An dversry hs complete control over network-level communiction to nd from nodes tht it controls. This cn compromise overlylevel communiction tht is routed through fulty node. Adversries my dely messges between correct nodes but we ssume tht ny messge sent by correct node to correct destintion over n overly route with no fulty nodes is delivered within time D with probbility P D. 2. Secure routing Net, we define secure routing primitive tht cn be combined with eisting techniques to construct secure pplictions on structured p2p overlys. Subsequent sections show how to implement the secure routing primitive under the fult nd network models tht we described in the previous section. The routing primitives implemented by current structured p2p overlys provide best-effort service to deliver messge to replic root ssocited with given key. With mlicious overly nodes, the messge my be dropped or corrupted, or it my be delivered to mlicious node insted of legitimte replic root. Therefore, these primitives cnnot be used to construct secure pplictions. For emple, when inserting n object, n ppliction cnnot ensure tht the replics re plced on legitimte, diverse replic roots s opposed to fulty nodes tht impersonte replic roots. Even if pplictions use cryptogrphic methods to uthenticte objects, mlicious nodes my still corrupt, delete, deny ccess to or supply stle copies of ll replics of n object. To ddress this problem, we define secure routing primitive. The secure routing primitive ensures tht when non-fulty node sends messge to key k, the messge reches ll non-fulty members in the set of replic roots R k with very high probbility. R k is defined s the set of nodes tht contins, for ech member of the set of replic keys ssocited with k, live root node tht is responsible for tht replic key. In Pstry, for instnce, R k is simply set of live nodes with nodeids numericlly closest to the key. Secure routing ensures tht (1) the messge is eventully delivered, despite nodes tht my corrupt, drop or misroute the messge; nd (2) the messge is delivered to ll legitimte replic roots for the key, despite nodes tht my ttempt to impersonte replic root. Secure routing cn be combined with eisting security techniques to sfely mintin stte in structured p2p overly. For instnce, self-certifying dt cn be stored on the replic roots, or Byzntine-fult-tolernt repliction lgorithm like BFT [4] cn be used to mintin the replicted stte. Secure routing gurntees tht the replics re initilly plced on legitimte replic roots, nd tht lookup messge reches replic if one eists. Similrly, secure routing cn be used to build other secure services, such s mintining file metdt nd user quots in distributed storge utility. The detils of such services re beyond the scope of this pper. Implementing the secure routing primitive requires the solution of three problems: securely ssigning nodeids to nodes, securely mintining the routing tbles, nd securely forwrding messges. Secure nodeid ssignment ensures tht n ttcker cnnot choose the vlue of nodeids ssigned to the nodes tht the ttcker controls. Without it, the ttcker could rrnge to control ll replics of given object, or to medite ll trffic to nd from victim node. Secure routing tble mintennce ensures tht the frction of fulty nodes tht pper in the routing tbles of correct nodes does not eceed, on verge, the frction of fulty nodes in the entire overly. Without it, n ttcker could prevent correct messge delivery, given only reltively smll number of fulty nodes. Finlly, secure messge forwrding ensures tht t lest one copy of messge sent to key reches ech correct replic root for the key with high probbility. Sections 3, 4 nd describe solutions to ech of these problems.

5 3 Secure nodeid ssignment The performnce nd security of structured p2p overly networks depend on the fundmentl ssumption tht there is uniform rndom distribution of nodeids tht cnnot be controlled by n ttcker. This section discusses wht goes wrong when the ttcker violtes this ssumption, nd how this problem cn be ddressed. 3.1 Attcks Attckers who cn choose nodeids cn compromise the integrity of structured p2p overly, without needing to control prticulrly lrge frction of the nodes. For emple, n ttcker my prtition Pstry or Chord overly if she controls two complete nd disjoint neighbor sets. Such ttckers my lso trget prticulr victim nodes by crefully choosing nodeids. For emple, they my rrnge for every entry in victim s routing tble nd neighbor set to point to hostile node in Chord overly. At tht point, the victim s ccess to the overly network is completely medited by the ttcker. Attckers who cn choose nodeids cn lso control ccess to trget objects. The ttcker cn choose the closest nodeids to ll replic keys for prticulr trget object, thus controlling ll replic roots. As result, the ttcker could delete, corrupt, or deny ccess to the object. Even when ttckers cnnot choose nodeids, they my still be ble to mount ll the ttcks bove (nd more) if they cn obtin lrge number of legitimte nodeids esily. This is known s Sybil ttck [1]. Previous pproches to nodeid ssignment hve either ssumed nodeids re chosen rndomly by the new node [] or compute nodeids by hshing the IP ddress of the node [2]. Neither pproch is secure becuse n ttcker hs the opportunity either to choose nodeids tht re not necessrily rndom, or to choose n IP ddress tht hshes to desired intervl in the nodeid spce. Prticulrly s IPv is deployed, even modest ttckers will hve more potentil IP ddresses t their disposl thn there re likely to be nodes in given p2p network. 3.2 Solution: certified nodeids One solution to securing the ssignment of nodeids is to delegte the problem to centrl, trusted uthority. We use set of trusted certifiction uthorities (CAs) to ssign nodeids to principls nd to sign nodeid certifictes, which bind rndom nodeid to the public key tht speks for its principl nd n IP ddress. The CAs ensure tht nodeids re chosen rndomly from the id spce, nd prevent nodes from forging nodeids. Furthermore, these certifictes give the overly public key infrstructure, suitble for estblishing encrypted nd uthenticted chnnels between nodes. Like conventionl CAs, ours cn be offline to reduce the risk of eposing certificte signing keys. They re not involved in the regulr opertion of the overly. Nodes with vlid nodeid certifictes cn join the overly, route messges, nd leve repetedly without involvement of the CAs. As with ny CA infrstructure, the CA s public keys must be well known, nd cn be instlled s prt of the node softwre itself, s is done with current Web browsers. The inclusion of n IP ddress in the certificte deserves some eplntion. Some p2p protocols, such s Tpestry nd Pstry, mesure the network dely between nodes nd choose routing tble entries tht minimize dely. If n ttcker with multiple legitimte nodeid certifictes could freely swp certifictes mong nodes it controls, it might be ble to increse the frction of ttcker s nodes in trget node s routing tble. By binding the nodeid to n IP ddress, it becomes hrder for n ttcker to move nodeids cross nodes. We llow multiple nodeid certifictes per IP ddress becuse the IP ddresses of nodes my chnge nd becuse otherwise, ttckers could deny service by hijcking victim s IP ddresses. A downside of binding nodeids to IP ddresses is tht, if node s IP ddress chnges, either s result of dynmic ddress ssignment, host mobility, or orgniztionl network chnges, then the node s old certificte nd nodeid become invlid. In p2p systems where IP ddresses re llowed to chnge dynmiclly, nodeid swpping ttcks my be unvoidble. Certified nodeids work well when nodes hve fied nodeids, which is the cse in Chord, Pstry, nd Tpestry. However, it might be hrder to secure nodeid ssignment in CAN. CAN nodeids represent zone in d- dimensionl spce tht is split in hlf when new node joins [1]. Both the nodeid of the originl node nd the nodeid of the joining node chnge during this process Sybil ttcks While nodeid ssignment by CA ensures tht nodeids re chosen rndomly, it is lso importnt to prevent n ttcker from esily obtining lrge number of nodeid certifictes. One solution is to require n ttcker to py money for certifictes, vi credit crd or ny other suitble mechnism. With this solution, the cost of n ttck grows nturlly with the size of the network. For emple, if nodeid certifictes cost $2, controlling 1% of n overly with 1, nodes costs $2, nd the cost rises to $2,, with 1,, nodes. The cost of trgeted ttcks is even higher; it costs n epected $2, to obtin the closest nodeid to prticulr point in the id spce in n overly with 1, nodes. Aprt from mking ttcks economiclly epensive, these fees cn lso fund the opertion of the CAs. Another solution is to bind nodeids to rel-world identities insted of chrging money. In prctice, different forms of CAs re suitble in different situtions.

6 The identity-bsed CA is the preferred solution in virtul privte overlys run by n orgniztion tht lredy mintins employment or membership records with strong identity checks. In n open Internet deployment, money-only CA my be more suitble becuse it voids the compleities of uthenticting rel-world identities. None of the known solutions to nodeid ssignment re effective when the overly network is very smll. For smll overly networks, we must require tht ll members of the network re trusted not to chet. Only when network reches criticl mss, where it becomes sufficiently hrd for n ttcker to muster enough resources to control significnt frction of the overly, should untrusted nodes be llowed to join. 3.3 Rejected: distributed nodeid genertion The CAs represent points of filure, vulnerble to both technicl nd legl ttcks. Also, for some p2p networks, it my be cumbersome to require users to spend money or prove their rel-world identities. Therefore, it would be desirble to construct secure p2p overlys without requiring centrlized uthorities, fees or identity checks. Unfortuntely, fully decentrlized nodeid ssignment ppers to hve fundmentl security limittions [1]. None of the methods we re wre of cn ultimtely prevent determined ttcker from cquiring lrge collection of nodeids. However, severl techniques my be ble to, t minimum, moderte the rte t which n ttcker cn cquire nodeids. One possible solution is to require prospective nodes to solve crypto puzzles [1] to gin the right to use nodeid, n pproch tht hs been tken to ddress number of denil of service ttcks [13, 8]. Unfortuntely, the cost of solving crypto puzzle must be cceptble to the slowest legitimte node, yet the puzzle must be hrd enough to sufficiently slow down n ttcker with ccess to mny fst mchines. This conflict limits the effectiveness of ny such technique. For completeness, we briefly describe here one reltively simple pproch to generte certified nodeids in completely distributed fshion using crypto puzzles. The ide is to require new nodes to generte key pir with the property tht the SHA-1 hsh of the public key hs the first p bits zero. The epected number of opertions required to generte such key pir is 2 p. The properties of public-key cryptogrphy llow the nodes to use secure hsh of the public key s their nodeid. This hsh should be computed using SHA-1 with different initiliztion vector or MD to void reducing the number of rndom bits in nodeids. Nodes cn prove tht they performed the required mount of work to use nodeid without reveling informtion tht would llow others to reuse their work. The vlue of p cn be set to chieve the desired level of security. It is lso possible to bind IP ddresses with nodeids to void ttcks on overlys tht eploit network loclity. The ide is to require nodes to consume resources in order to be ble to use given nodeid with n IP ddress. We do this by requiring nodes to find string such tht SHA-1(SHA-1(ipddr,),nodeId) hs p bits equl to zero. Nodes would be required to present such n for the pir (nodeid,ipddr) to be ccepted by others. Finlly, it is possible to periodiclly invlidte nodeids by hving some trusted entity brodcst to the overly messge supplying different initiliztion vector for the hsh computtions. This mkes it hrder for n ttcker to ccumulte mny nodeids over time nd to reuse nodeids computed for one overly in nother overly. However, it requires legitimte nodes to periodiclly spend dditionl time nd communiction to mintin their membership in the overly. 4 Secure routing tble mintennce We now turn our ttention to the problem of secure routing tble mintennce. The routing tble mintennce mechnisms re used to crete routing tbles nd neighbor sets for joining nodes, nd to mintin them fter cretion. Idelly, ech routing tble nd neighbor set should hve n verge frction of only f rndom entries tht point to nodes controlled by the ttcker (clled bd entries ). But ttckers cn increse the frction of bd entries by supplying bd routing updtes, which reduces the probbility of routing successfully to replic roots. Preventing ttckers from choosing nodeids is necessry to void this problem but it is not sufficient s illustrted by the two ttcks discussed net. We lso discuss solutions to this problem. 4.1 Attcks The first ttck is imed t routing lgorithms tht use network proimity informtion to improve routing efficiency: ttckers my fke proimity to increse the frction of bd routing tble entries. For emple, the network model tht we ssumed llows n ttcker to control communiction to nd from fulty nodes tht it controls. When correct node p sends probe to estimte dely to fulty node with certin nodeid, n ttcker cn intercept the probe nd hve the fulty node closest to p reply to it. If the ttcker controls enough fulty nodes spred over the Internet, it cn mke nodes tht it controls pper close to correct nodes to increse the probbility tht they re used for routing. The ttck is hrder when c (the miml frction of colluding nodes) is smll even if f is lrge. This ttck cn be ruled out by more restrictive communiction model, since nodeid certifictes bind IP ddresses to nodeids (see Section 3.2). For emple, if fulty nodes cn only observe messges tht re sent to

7 their own IP ddress [19], this ttck is prevented. But note tht rogue ISP or corportion with severl offices round the world could esily perform this ttck by configuring their routers ppropritely. The ttck is lso possible if there is ny other form of indirection tht the ttcker cn control, e.g., mobile IPv. The second ttck does not mnipulte proimity informtion. Insted, it eploits the fct tht it is hrd to determine whether routing updtes re legitimte in overly protocols like Tpestry nd Pstry. Nodes receive routing updtes when they join the overly nd when other nodes join, nd they fetch routing tble rows from other nodes in their routing tble periodiclly to ptch holes nd reduce hop delys. In these systems, ttckers cn more esily supply routing updtes tht lwys point to fulty nodes. This simple ttck cuses the frction of bd routing tble entries to increse towrd one s the bd routing updtes re propgted. More precisely, routing updtes from correct nodes point to fulty node with probbility t lest f wheres this probbility cn be s high s one for routing updtes from fulty nodes. Correct nodes receive updtes from other correct nodes with probbility t most 1 f nd from fulty nodes with probbility t lest f. Therefore, the probbility tht routing tble entry is fulty fter n updte is t lest (1 f ) f + f 1, which is greter thn f. This effect cscdes with ech subsequent updte, cusing the frction of fulty entries to tend towrds one. Systems without strong constrints on the set of nodeids tht cn fill ech routing tble slot re more vulnerble to this ttck. Pstry nd Tpestry impose very wek constrints t the top levels of routing tbles. This fleibility mkes it hrd to determine if routing updtes re unbised but it llows these systems to effectively eploit network proimity to improve routing performnce. CAN nd Chord impose strong constrints on nodeids in routing tble entries: they need to be the closest nodeids to some point in the id spce. This mkes it hrd to eploit network proimity to improve performnce but it is good for security; if ttckers cnnot choose the nodeids they control, the probbility tht n ttcker controls the nodeid closest to point in the id spce is f. 4.2 Solution: constrined routing tble To enble secure routing tble mintennce, it is importnt to impose strong constrints on the set of nodeids tht cn fill ech slot in routing tble. For emple, the entry in ech slot cn be constrined to be the closest nodeid to some point in the id spce s in Chord. This constrint cn be verified nd it is independent of network proimity informtion, which cn be mnipulted by ttckers. The solution tht we propose uses two routing tbles: one tht eploits network proimity informtion for efficient routing (s in Pstry nd Tpestry), nd one tht constrins routing tble entries (s in Chord). In norml opertion, the first routing tble is used to forwrd messges to chieve good performnce. The second one is used only when the efficient routing technique fils. We use the test in Section.2 to detect when routing fils. We modified Pstry to use this solution. We use the norml loclity-wre Pstry routing tble nd n dditionl constrined Pstry routing tble. In the loclity-wre routing tble of node with identifier i, the slot t level l nd domin d cn contin ny nodeid tht shres the first l digits with i nd hs the vlue d in the l + 1st digit. In the constrined routing tble, the entry is further constrined to point to the closest nodeid to point p in the domin. We define p s follows: it shres the first l digits with i, it hs the vlue d in the l +1st digit, nd it hs the sme remining digits s i. Pstry s messge forwrding works with the constrined routing tble without modifictions. The sme would be true with Tpestry. But the lgorithms to initilize nd mintin the routing tble were modified s follows. All overly routing lgorithms rely on bootstrp node to initilize the routing stte of newly joining node. The bootstrp node is responsible for routing messge using the nodeid of the joining node s the key. If the bootstrp node is fulty, it cn completely corrupt the view of the overly network s seen by the new node. Therefore, it is necessry to use set of diverse bootstrp nodes lrge enough to ensure tht with very high probbility, t lest one of them is correct. The use of nodeid certifictes mkes the tsk of choosing such set esier becuse the ttcker cnnot forge nodeids. A newly joining node, n, picks set of bootstrp nodes nd sks ll of them to route using its nodeid s the key. Then, non-fulty bootstrp nodes use secure forwrding techniques (described in Section.2) to obtin the neighbor set for the joining node. Node n collects the proposed neighbor sets from ech of the bootstrp nodes, nd picks the closest live nodeids from ech proposed set to be its neighbor set (where the definition of closest is protocol specific). The loclity-wre routing tble is initilized s before by collecting rows from the nodes long the route to the nodeid. The difference is tht there re severl routes; n picks the entry with miniml network dely from the set of cndidtes it receives for ech routing tble slot. Ech entry in the constrined routing tble cn be initilized by using secure forwrding to obtin the live nodeid closest to the desired point p in the id spce. This is similr to wht is done in Chord. The problem is tht it is quite epensive with b > 1 (recll tht b controls the number of columns in the routing tble of Tpestry nd Pstry). To reduce the overhed, we cn tke dvntge of the fct tht, by induction, the constrined routing tbles of the nodes in n s neighbor set point to entries tht

8 re close to the desired point p. Therefore, n cn collect routing tbles from the nodes in its neighbor set nd use them to initilize its constrined routing tble. From the set of cndidtes tht it receives for ech entry, it picks the nodeid tht is closest to the desired point for tht entry. As side effect of this process, n informs the nodes in its neighbor set of its rrivl. We eploit the symmetry in the constrined routing tble to inform nodes tht need to updte their routing tbles to reflect n s rrivl: n checks its neighbor set nd the set of cndidtes for ech entry to determine which cndidtes should updte routing tble entries to point to n. It informs those cndidtes of its rrivl. To ensure neighbor set stbiliztion in the bsence of new joins nd leves, n informs the members of its neighbor set whenever it chnges nd it periodiclly retrnsmits this informtion until its receipt is cknowledged. Secure messge forwrding The use of certified nodeids nd secure routing tble mintennce ensure tht ech constrined routing tble (nd neighbor set) hs n verge frction of only f rndom entries tht point to nodes controlled by the ttcker. But routing with the constrined routing tble is not sufficient becuse the ttcker cn reduce the probbility of successful delivery by simply not forwrding messges ccording to the lgorithm. The ttck is effective even when f is smll, s we will show. This section describes n efficient solution to this problem..1 Attcks All structured p2p overlys provide primitive to send messge to key. In the bsence of fults, the messge is delivered to the root node for the key fter n verge of h routing hops. But routing my fil if ny of the h 1 nodes long the route between the sender nd the root re fulty; fulty nodes my simply drop the messge, route the messge to the wrong plce, or pretend to be the key s root. Therefore, the probbility of routing successfully between two correct nodes when frction f of the nodes is fulty is only: (1 f ) h 1, which is independent of c. The root node for key my itself be fulty. As discussed before, pplictions cn tolerte root fults by replicting the informtion ssocited with the key on severl nodes the replic roots. Therefore, the probbility of routing successfully to correct replic root is only: σ = (1 f ) h. The vlue of h depends on the overly: it is (d/4)(n 1/d ) in CAN, log 2 (N)/2 in Chord, nd log 2 b(n) in Pstry nd Tpestry. We rn simultions of Pstry to vlidte this model. The model predicts probbility of success slightly lower thn the probbility tht we observed in the simultions (becuse the number of Pstry hops is slightly less thn prob. of successful routing N=1 N=1 N=1 N= percentge of nodes compromised Figure 3: Probbility of routing to correct replic. log 2 b(n) on verge [3]) but the error ws below 2%. Figure 3 plots the probbility of routing to correct replic in Pstry (computed using the model) for different vlues of f, N, nd b = 4. The probbility drops quite fst when f or N increse. Even with only 1% of the nodes compromised, the probbility of successful routing is only % when there re 1, nodes in Pstry overly. In CAN, Pstry, nd Tpestry, pplictions cn reduce the number of hops by incresing the vlue of d or b. Fewer hops increse the probbility of routing correctly. For emple, the probbility of successful delivery with f =.1 nd 1, nodes is % in Pstry when b = 4 nd 7% when b =. But incresing b lso increses the cost of routing tble mintennce; high probbility of routing success requires n imprcticlly lrge vlue of b. Chord currently uses fied b = 1, which results in low probbility of success, e.g., the probbility is only 42% under the sme conditions..2 Solution: detect fults, use diverse routes The results in Figure 3 show tht it is importnt to devise mechnisms to route securely. We wnt secure routing primitive tht tkes messge nd destintion key nd ensures tht with very high probbility t lest one copy of the messge reches ech correct replic root for the key. The question is how to do this efficiently. Our pproch is to route messge efficiently nd to pply filure test to determine if routing worked. We only use more epensive redundnt routing when the filure test returns positive. In more detil, our secure routing primitive routes messge efficiently to the root of the destintion key using the loclity-wre routing tble. Then, it collects the prospective set of replic roots from the prospective root node nd pplies the filure test to the set. If the test is negtive, the prospective replic roots re ccepted s the correct ones. If it is positive, messge copies re sent over diverse routes towrd the vrious replic roots such tht with high probbility ech correct replic root is reched. We strt by describ-

9 ing how to implement the filure test. Then we eplin redundnt routing nd why we rejected n lternte pproch clled itertive routing..2.1 Routing filure test The filure test tkes key nd set of prospective replic roots for the key. It returns negtive if the set of roots is likely to be correct for the key. Otherwise, it returns positive. Of course, routing cn fil without the sender ever receiving set of prospective replic roots. The sender detects this by strting timer when it sends messge. If it does not receive response before the timer epires, the filure test returns positive triggering the use of redundnt routing. Detecting routing filures is difficult becuse colition of fulty nodes cn pretend to be the legitimte replic roots for given key. Since the replic roots re determined by the structure of the overly, node whose nodeid is fr from the key must rely on overly routing to determine the correct set of replic roots. But if messge is routed by fulty node, the dversry cn fbricte credible route nd replic root set tht contin only nodes it controls. Furthermore, it might be the cse tht the dversry just hppens to legitimtely control one of the ctul replic roots. This problem is common to ll structured p2p overly protocols. The routing filure test is bsed on the observtion tht the verge density of nodeids per unit of volume in the id spce is greter thn the verge density of fulty nodeids. The test works by compring the density of nodeids in the neighbor set of the sender with the density of nodeids close to the replic roots of the destintion key. We describe the test in detil only in the contet of Pstry to simplify the presenttion; the generliztion to other overlys is strightforwrd. Overlys tht distribute replic keys for key uniformly over the id spce cn still use this check by compring the density t the sender with the verge distnce between ech replic key nd its root s nodeid. In Pstry, the set of replic roots for key is subset of the neighbor set of the key s root node, clled the key s root neigbor set. Ech correct node p computes the verge numericl distnce, µ p, between consecutive nodeids in its neighbor set. The neighbor set of p contins l + 1 live nodes: p, the l/2 nodes with the closest nodeids less thn p s, nd the l/2 nodes with the closest nodeids greter thn p s. To test prospective root neighbor set, rn = id,...,id l+1, for key, p checks tht: 1. ll nodeids in rn hve vlid nodeid certificte, the closest nodeid to the key is the middle one, nd the nodeids stisfy the definition of neighbor set 2. the verge numericl distnce, µ rn, between consecutive nodeids in rn stisfies: µ rn < µ p γ If rn stisfies both conditions, the test returns negtive; otherwise, it returns positive. The test cn be inccurte in one of two wys: it cn return flse positive when the prospective root neighbor set is correct, or it cn return flse negtive when the prospective set is incorrect. We cll the probbility of flse positives α nd the probbility of flse negtives β. The prmeter γ controls the trdeoff between α nd β. Intuitively, incresing γ decreses α but it lso increses β. Assuming tht there re N live nodes with nodeids uniformly distributed over the id spce (which hs length D = ), the distnces between consecutive nodeids re pproimtely independent eponentil rndom vribles with men D/N for lrge N. The sme holds for the distnces between consecutive nodeids of fulty nodes tht cn collude together but the men is D/(c N). It is interesting to note tht α nd β re independent of f. They only depend on the upper bound, c, on the frction of colluding nodes becuse fulty nodes only know the identities of fulty nodes tht they collude with. Under these ssumptions, we hve derived the following epressions to compute α nd β (see detiled derivtion in the Appendi): α(n,k,γ) = nn k k e n k (n 1)!(k 1)! u n 1 e n(u 1) (n 1)! β(n,k,γ,c) = α(k,n, 1 γc ) γu v k 1 e k(v 1) dvdu (k 1)! These epressions cn be used to compute α nd β numericlly. We lso computed the following closed-form upper bounds for α nd β: { [ α ep k (r + 1)log r + γ ]} logγ r + 1 { [ β ep k (r + 1)log r + γc ]} + log(γc) r + 1 where n is the number of distnce smples used to compute µ p, k is the number of distnce smples used to compute µ rn, nd r = n/k. The test bove used n = k = l. The nlysis shows tht α nd β re independent of N (provided k N), nd tht the test s ccurcy cn be improved by incresing the number of distnce smples used to compute the mens. It is esy to increse the number of smples n used to compute µ p by ugmenting the mechnism tht is lredy in plce to stbilize neighbor sets. This mechnism propgtes nodeids tht re dded nd removed from neighbor set to the other members of the set; it cn be etended to propgte nodeids further but we omit the detils due to lck of spce. It is hrd to increse the number of smples used to compute µ rn becuse of some ttcks tht we describe below. Therefore, we keep k = l. We rn severl simultions to evlute the effectiveness of our routing filure test. The simultions rn in system with 1, rndom nodeids. Figure 4 plots vlues of α nd β for different vlues of γ with f = c =.3, the

10 lph nd bet bet upper bound bet predicted bet mesured lph mesured lph predicted lph upper bound gmm Figure 4: Routing filure test: probbility of flse positives (α) nd negtives (β). The predicted curves re lmost indistinguishble from the simultion mesurements but the upper bounds re not tight. number of smples t the sender is n = 2, nd the number of root neighbors is k = l = 32. The figure shows predicted vlues computed numericlly, the upper bounds, nd vlues mesured in the simultions. The predicted curves mtch the mesured curves lmost ectly but the upper bounds re not very tight. The minimum error is obtined when α = β, which is equl to.8 with γ = 1.72 in this cse. Attcks: There re severl ttcks tht could invlidte the nlysis nd weken our routing filure test. First, the ttcker cn collect nodeid certifictes of nodes tht hve left the overly, nd use them to increse the density of prospective root neighbor set. Second, the ttcker cn include both nodeids of nodes it controls nd nodeids of correct nodes in prospective root neighbor set. Both ttcks cn reduce the probbility tht messges rech ll correct replic roots. The second ttck is hrder to counter in overlys tht distribute replic keys over the id spce becuse replic roots hve no detiled knowledge bout the nodeids close to other replic keys. These ttcks cn be voided by hving the sender contct ll the prospective root neighbors to determine if they re live nd if they hve nodeid certificte tht ws omitted from the prospective root neighbor set. To implement this efficiently, the prospective root returns to the sender messge with the list of nodeid certifictes, list with the secure hshes of the neighbor sets reported by ech of the prospective root neighbors, nd the set of nodeids (not in the prospective root neighbor set) tht re used to compute the hshes in this list. The sender checks tht the hshes re consistent with the identifiers of the prospective root neighbors. Then, it sends ech prospective root neigbor the corresponding neighbor set hsh for confirmtion. In the bsence of fults, the root neighbors will confirm the hshes nd the sender cn perform the density com- lph = bet percentge of nodes compromised Figure : Routing filure test: minimum error probbility without nodeid suppression ttcks nd vrying number of smples. prison immeditely. For sufficiently lrge timeout, this hppens with probbility τ = binom(; k, f ), where binom is the binomil distribution [] nd k is the number of root neighbors. With fulty nodes in the prospective root neighbor set, the routing filure test my require more communiction before the density check cn be run. We re still studying the best strtegy to del with this cse. Currently, we consider the test filed when the prospective root neighbors don t gree nd use redundnt routing. But, it my be worthwhile investing some dditionl communiction before reverting to redundnt routing. In ddition to these ttcks, there is nodeid suppression ttck tht seems to be unvoidble nd significntly decreses the ccurcy of this test. The ttcker cn suppress nodeids close to the sender by leving the overly, which increses β. Similrly, the ttcker cn suppress nodeids in the root neighbor set, which increses α. Furthermore, the ttcker cn lternte between the two modes nd honest nodes hve no wy of detecting in which mode they re operting. We rn simultions to compute the minimum error probbility (α = β) with nd without nodeid suppression ttcks for different vlues of c = f. The probbility of error increses fst with c nd it is higher thn.1 for c.3 even with 2 smples t the sender. The nodeid suppression ttck increses the minimum probbility of error for lrge percentges of compromised nodes, e.g., the probbility of error is higher thn.1 for c.2 even with 2 smples t the sender. Figures nd show the results without nd with nodeid suppression ttcks, respectively. These results indicte tht our routing filure test is not very ccurte. But, fortuntely we cn trde off n increse in α to chieve trget β nd use redundnt routing to dismbigute flse positives. We rn simultions to determine the minimum α tht cn be chieved for trget β =.1 with different vlues of c = f, nd different numbers of smples t the sender. Figure 7 shows

11 lph = bet percentge of nodes compromised Figure : Routing filure test: minimum error probbility with nodeid suppression ttcks nd vrying number of smples. lph with bet= percentge of nodes compromised Figure 7: Routing filure test: probbility of flse positives for flse negtive rte of.1 with nodeid suppression ttcks nd vrying number of smples. the results with nodeid suppression ttcks. The results show tht the test is not meningful for this trget β nd c >.3 with nodeid suppression ttcks. However, setting γ = 1.23 with 2 smples t the sender enbles the routing filure test to chieve the trget β for c.3. For this vlue of γ nd with c =.3, nodeid suppression ttcks cn increse α to.77. But without nodeid suppression ttcks the vlue of α is only.12, i.e., redundnt routing is required 12% of the time..2.2 Redundnt routing The redundnt routing technique is invoked when the routing filure test is positive. The ide is simply to route copies of the messge over multiple routes towrd ech of the destintion key s replic roots. If enough copies of the messge re sent long diverse routes to ech replic key, ll correct replic roots will receive t lest one copy of the messge with high probbility. The issue is how to ensure tht routes re diverse. One pproch is to sk the members of the sender s neighbor set to forwrd the copies of the messge to the replic keys. This technique is sufficient in overlys tht distribute the replic keys uniformly over the id spce (e.g., CAN nd Tpestry). But it is not sufficient in overlys tht choose replic roots in the neighbor set of the key s root (e.g., Chord nd Pstry) becuse the routes ll converge on the key s root, which might be fulty. For these overlys, we developed technique clled neighbor set nycst tht sends copies of the messge towrd the destintion key until they rech node with the key s root in its neighbor set. Then it uses the detiled knowledge tht such node hs bout the portion of the id spce round the destintion key to ensure tht ll correct replic roots receive copy of the messge. To simplify the presenttion, we only describe in detil how redundnt routing works in Pstry. If correct node p sends messge to destintion key nd the routing filure test is positive, it does the following: (1) p sends r messges to the destintion key. Ech messge is forwrded vi different member of p s neighbor set; this cuses the messges to use diverse routes. All messges re forwrded using the constrined routing tble nd they include nonce. (2) Any correct node tht receives one of the messges nd hs s root in its neighbor set returns its nodeid certificte nd the nonce, signed with its privte key, to p. (3) p collects in set N the l/2 + 1 nodeid certifictes numericlly closest to on the left, nd the l/2 + 1 closest to on the right. Only certifictes with vlid signed nonces re dded to N nd they re first mrked pending. (4) After timeout or fter ll r replies re received, p sends list with the nodeids in N to ech node mrked pending in N nd mrks the nodes done. () Any correct node tht receives this list forwrds p s originl messge to the nodes in its neighbor set tht re not in the list, or it sends confirmtion to p if there re no such nodes. This my cuse steps 2 to 4 to be repeted. () Once p hs received confirmtion from ech of the nodes in N, or step 4 ws eecuted three times, it computes the set of replic roots for from N. If the timeout is sufficiently lrge nd correct nodes hve nother correct node in ech hlf of their neighbor set 1, the probbility of reching ll correct replic roots of is pproimtely equl to the probbility tht t lest one of the nycst messges is forwrded over route with no fults to correct node with the key s root in its neighbor set. Assuming independent routes, this probbility is: 1 binom(;r,(1 f ) 1+log 2 bn ) where binom is the binomil distribution [] with successful routes, r trils, nd the probbility of routing successfully in ech tril is (1 f ) 1+log 2 b N. The +1 counts 1 The neighbor set size l should be chosen to ensure this with high probbility

12 the etr hop for messges routed through neighbor set member. The probbility of success for this technique depends on f nd is independent of c. We lso rn simultions to determine the probbility of reching ll correct replic roots with our redundnt routing technique. Figure 8 plots the predicted probbility nd the probbility mesured in the simultor for 1, nodes, b = 4, nd l = r = 32. The nlytic model mtches the results well for high success probbilities. The results show tht the probbility of success is greter thn.999 for f <.3. Therefore, this technique combined with our routing filure test cn chieve relibility of pproimtely.999 for f <.3. prob. of successful routing percentge of nodes compromised mesured predicted Figure 8: Model nd simultion results for the probbility of reching ll correct replic roots using redundnt routing with neighbor set nycst. We studied severl versions of redundnt routing tht chieve the sme probbility of success but perform differently. For emple, the signed nonces re used to ensure tht the nodeid certifictes in N belong to live nodes. But nodes cn void signing nonces by periodiclly signing their clock reding in system with loosely synchronized clocks, nd no signtures re necessry if the ttcker cnnot forge IP source ddresses. We re still eploring the design spce. For emple, it should be possible to improve performnce significntly by sending the nycst messges one t time nd using version of the routing filure test fter ech one. This pproch would lso work well when reding selfcertifying dt..2.3 Putting it ll together: performnce The performnce of Pstry s secure routing primitive depends on the cost of the routing filure test, the cost of redundnt routing, nd on the probbility tht redundnt routing cn be voided. This section presents n nlysis of these costs nd probbility. For simplicity, we ssume tht ll fulty nodes cn collude (c = f ), the number of probes used by redundnt routing is equl to the neighbor set size (r = l), the number of smples t the source for routing filure tests is n = 2, nd the number of nodes in the overly is N = 1,. The cost of the routing filure test when it returns negtive is n etr round-trip dely nd 2l +1 messges. The totl number of bytes in ll messges is: l (IdSize + 2HshSize) + (l + 1) IdCertSize + (2l + 1) HdrSize Using PSS-R [1] for signing nodeid certifictes with 124-bit modulus nd 12-bit modulus for the node public keys, the nodeid certificte size is 128B. Therefore, the etr bndwidth consumed by the routing filure test is pproimtely. KB with l = 32 nd 2.9 KB with l = 1 (plus the spce used up by messge heders). When the test returns positive, it dds the sme number of messges nd bytes but the etr dely is the timeout period. The cost of redundnt routing depends on the vlue of f. The best cse occurs when ll of the root s neighbor set is dded to N in the first itertion. In this cse, redundnt routing dds log 2 b N + 3 etr messge delys nd l (log 2 b N + 3) messges. The totl number of bytes in these messges is: l (l IdSize + IdCertSize + SigSize) + l (log 2 b N + 3) HdrSize Using PSS-R for signing nonces, the signed nonce size is 4B. Therefore, the etr bndwidth consumed in this cse is 22 KB with l = 32 nd 7 KB with l = 1 (plus the spce used up by messge heders). Under ttck redundnt routing dds dely of t most three timeout periods nd the epected number of etr messges is less thn l (log 2 b N +2)+(l g) (3+g), where g = l (1 f ) log 2 b N+1 is the epected number of correct nodes in the root s neighbor set tht is dded to N in the first itertion. The epected number of messges is less thn 41 with l = 32 nd f =.2 nd less thn 188 with l = 1 nd f =.18. The totl number of bytes sent under ttck is similr to the best cse vlue ecept tht the sender sends n dditionl l(l g) IdSize bytes in nodeid lists nd the number of messges increses. This is n dditionl 12 KB with l = 32 nd f =.2 nd 1 KB with l = 1 nd f =.18 (plus the spce used up by messge heders). The probbility of voiding redundnt routing is given by σ τ (1 α), where σ is the probbility tht the overly routes the messge to the correct replic root, τ is the probbility tht there re no fulty nodes in the neighbor set of the root, nd α is the flse positive rte of the routing filure test. We use σ = (1 f ) log 2 b N, which ssumes tht routing tbles hve n verge of f rndom bd entries. This ssumption holds for the loclity-wre routing tble in the bsence of the ttcks discussed in Section 4 nd it holds for the constrined routing tble even with these ttcks. We do not hve good model of the effect of these ttcks on the loclity wre routing tble but we believe tht they re very hrd to mount for smll vlues of f (.1).

13 prob. no redundnt routing scenrio 1 scenrio percentge of nodes compromised Figure 9: Probbility of voiding redundnt routing in two scenrios: (1) f.18 Σ.999 with γ = 1.8 nd l = 1, nd (2) f.2 Σ.999 with γ = 1.8 nd l = 32. The prmeters γ nd l, should be set bsed on the desired security level, which cn be epressed s the probbility Σ tht ll correct replic roots receive copy of the messge. The overly size nd the ssignment of vlues to the prmeters implicitly define bound on f. If this bound is eceeded, Σ will drop. For emple, we sw tht f.3 Σ.999 with γ = 1.23 nd l = 32. But redundnt routing is invoked 12% of the time for this vlue of γ even with no fults. One cn trde off security for improved performnce by incresing γ to reduce α, nd by decresing l to reduce the cost of the routing filure test nd redundnt routing nd to increse τ. For emple, consider the following two scenrios: (1) f.18 Σ.999 with γ = 1.8 nd l = 1, nd (2) f.2 Σ.999 with γ = 1.8 nd l = 32. Figure 9 plots the probbility of voiding redundnt routing in these two scenrios for different vlues of f. Without fults, redundnt routing is invoked only.% of the time in scenrio (1) nd.4% in (2). In the common cse when the frction of fulty nodes is smll, the routing filure test improves performnce significntly by voiding the cost of redundnt routing..2.4 Rejected: checked itertive routing An lterntive to redundnt routing is itertive routing, s suggested in Sit nd Morris [19]: the sender strts by looking up the net hop in its routing tble nd setting vrible n to point to this node; then, the sender sks n for the net hop nd updtes n to point to the returned vlue. The process is repeted until this vlue is the root of the destintion key. Itertive routing doubles the cost reltive to the more trditionl recursive solution but it my increse the probbility of routing successfully becuse it llows the sender to pick n lterntive net hop when it fils to receive n entry from node. This is not strong defense ginst n ttcker who cn provide fulty node s the net hop. However, itertive routing cn be ugmented with hop tests to check whether the net hop in route is correct. Hop tests re effective in systems like Chord or Pstry with the constrined routing tble becuse ech routing tble entry should contin the nodeid closest to specific point p in the id spce. One cn use mechnism identicl to the nodeid density checking tht we used for the routing filure test. The only difference is tht the verge distnce between consecutive nodeids close to the sender is compred to the distnce between the nodeid in the routing tble entry nd the desired point p. We rn simultions to compute the flse positive nd flse negtive rtes for this pproch with different vlues of c (these rtes re independent of f ). For emple, the minimum error for this hop test (α = β) is equl to pproimtely.3 with c =.3 nd 2 smples to compute the men t the sender. The error is high becuse there is single smple t the destintion hop. However, our simultions indicte tht itertive lookups using Pstry s constrined routing tble with this hop check improve the probbility of routing successfully. For emple, the probbility of routing successfully with c =.3, N = 1,, b = 4, l = 32, nd 2 smples to compute the men t the sender, improves from below.3 to.4. But it dds n etr 2.7 hops to ech route on verge becuse of flse positives. We tried to increse the number of smples by hving the sender fetch n entire routing tble row during ech itertive routing step without reveling the inde of the required slot. Unfortuntely, this performs worse thn obtining single smple becuse the ttcker cn combine good nd bd routing tble entries to obtin high verge density. We lso tried to combine checked itertive routing with the redundnt routing technique tht we described before. We used checked itertive routing for the neighbor set nycst messges in the hope tht the improved success probbility for the itertive routes would result in n improvement over redundnt routing with recursive routes. But there ws no visible improvement, most likely becuse the itertive routes re less independent thn the recursive routes. We conclude tht the routing filure test combined with redundnt routing is the most effective solution for implementing secure routing. Self-certifying dt The secure routing primitive dds significnt overhed over conventionl routing. In this section, we describe how the use of secure routing cn be minimized by using self-certifying dt. The relince on secure routing cn be reduced by storing self-certifying dt in the overly, i.e., dt whose integrity cn be verified by the client. This llows clients to use efficient routing to request copy of n object.

14 If client receives copy of the object, it cn check its integrity nd resort to secure routing only when the integrity check fils or there ws no response within timeout period. Self-certifying dt does not help when inserting new objects in the overly or when verifying tht n object is not stored in the overly. In these cses, we use the secure routing primitive to ensure tht ll correct replic roots re reched. Similrly, node joining requires secure routing. Nevertheless, self-certifying dt cn eliminte the overhed of secure routing in common cses. Self-certifying dt hs been used in severl systems. For emple, CFS [7] uses cryptogrphic hsh of file s contents s the key during insertion nd lookup of the file, nd PAST [18] inserts signed files into the overly. The technique cn be etended to support mutble objects with strong consistency gurntees. One cn use system like PAST to store self-certifying group descriptors tht identify the set of hosts responsible for replicting the object. Group descriptors cn be used s follows. At object cretion time, the owner of the object uses secure routing to insert group descriptor into the overly under key tht identifies the object. The descriptor contins the public keys nd IP ddresses of the object s replic holders nd it is signed by the owner. The replic group cn run Byzntine-fult-tolernt repliction lgorithm like BFT [4] nd the initil group membership is the set of replic roots ssocited with the key. In this setting, red nd write opertions cn be performed s follows: the client uses efficient routing to retrieve group descriptor from the overly nd checks the descriptor s signture; if correct, it uses the informtion in the descriptor to uthenticte the replic holders nd to invoke replicted opertion. If the client fils to retrieve vlid descriptor or if it fils to uthenticte the replic holders, it uses the secure routing primitive to obtin correct group descriptor or to ssert tht the object does not eist. This procedure provides strong consistency gurntees (linerizbility [11]) for reds nd writes while voiding the routing filure test in the common cse. Chnging the membership of the group tht is responsible for replicting n object is not trivil; it requires securely inserting new group descriptor in the overly nd ensuring tht clients cn relibly detect stle group descriptors. The following technique llows groups to chnge membership while retining strong consistency gurntees. Ech group of hosts tht stores replics of given object mintins privte/public key pir ssocited with the group. When the group membership chnges, ech host in the new membership genertes new key pir for the group, the hosts in the old membership use their old keys to sign new group descriptor contining the new keys, nd then delete the old keys. If this opertion is performed by quorum of replic holders before the bound on the number of fulty group members is eceeded [4], old replic holders tht fil will not be ble to collude to pretend they re the current group becuse they cnnot form the quorum necessry to uthenticte themselves to client. Group descriptors cn be uthenticted by following signture chin tht strts with n owner signture nd hs signtures of quorum of replics for ech subsequent membership chnge. The chin cn be shortened by new signture from the owner or, lterntively, replics cn use proctive signture shring [12] to void the need for chining signtures. 7 Relted work Sit nd Morris [19] present frmework for performing security nlyses of p2p networks. Their dversril model llows for nodes to generte pckets with rbitrry contents, but ssumes tht nodes cnnot intercept rbitrry trffic. They then present tonomy of possible ttcks. At the routing lyer, they identify node lookup, routing tble mintennce, nd network prtitioning / virtuliztion s security risks. They lso discuss issues in higher-level protocols, such s file storge, where nodes my not necessrily mintin the necessry invrints, such s storge repliction. Finlly, they discuss vrious clsses of denil-of-service ttcks, including rpidly joining nd leving the network, or rrnging for other nodes to send bulk volumes of dt to overlod victim s network connection (i.e., distributed denil of service ttcks). Dingledine et l. [9] nd Douceur [1] discuss ddress spoofing ttcks. With lrge number of potentilly mlicious nodes in the system nd without trusted centrl uthority to certify node identities, it becomes very difficult to know whether you cn trust the climed identity of somebody to whom you hve never before communicted. Dingledine proposes to ddress this with vrious schemes, including the use of micro-csh, tht llow nodes to build up reputtions. Bellovin [2] identifies number of issues with Npster nd Gnutell. He discusses how difficult it might be to limit Npster nd Gnutell use vi firewlls, nd how they cn lek informtion tht users might consider privte, such s the serch queries they issue to the network. Bellovin lso epresses concern over Gnutell s push feture, intended to work round firewlls, which might be useful for distributed denil of service ttcks. He considers Npster s centrlized rchitecture to be more secure ginst such ttcks, lthough it requires ll users to trust the centrl server. It is worthwhile mentioning very elegnt lterntive solution for secure routing tble mintennce nd forwrding tht we rejected. This solution replces ech node

15 by group of diverse replics s suggested by Lynch et l. [14]. The replics re coordinted using stte mchine repliction lgorithm like BFT [4] tht cn tolerte Byzntine fults. BFT cn replicte rbitrry stte mchines nd, therefore, it cn replicte Pstry s routing tble mintennce nd forwrding protocols. Additionlly, the lgorithm in [14] provides strong consistency gurntees for overly routing nd mintennce. However, there re two disdvntges: the solution is epensive even without fults, nd it is less resilient thn the solution tht we propose. Ech routing step is epensive becuse it requires n greement protocol between the replics. Since the replics should be geogrphiclly dispersed to reduce the probbility of correlted fults, greement ltency will be high. Additionlly, ech group of replics must hve less thn 1/3 of its nodes fulty. This bound on the number of fulty replics per group results in reltively low probbility of successful routing. The probbility tht replic group with r replics is correct when frction f of the nodes in the Pstry overly is compromised is r/3 i= binom(i;r, f ), where binom denotes the binomil distribution with i successes, r trils, nd probbility of success f. For emple, the probbility tht replic group is correct with 2% of the nodes compromised nd 32 replics is less thn 93%. In this emple, the probbility of routing correctly with 1, nodes in the overly is only 72%. 8 Conclusions Structured peer-to-peer overly networks hve previously ssumed fil-stop model for nodes; ny node ccessible in the network ws ssumed to correctly follow the protocol. However, if nodes re mlicious nd conspire with ech other, it is possible for smll number of nodes to compromise the overly nd the pplictions built upon it. This pper hs presented the design nd nlysis of techniques for secure node joining, routing tble mintennce, nd messge forwrding in structured p2p overlys. These techniques provide secure routing, which cn be combined with eisting techniques to construct pplictions tht re robust in the presence of mlicious prticipnts. A routing filure test llows the use of efficient proimity-wre routing in the common cse, resorting to the more costly redundnt routing technique only when the test indictes possible interference by n ttcker. Moreover, we show how the use of secure routing cn be reduced by using self-certifying ppliction dt. These techniques llow us to tolerte up to 2% mlicious nodes while providing good performnce when the frction of compromised nodes is smll. Acknowledgments We wish to thnk Robert Morris, Rodrigo Rodrigues, Fbien Petitcols, our shepherd Dvid Wetherll nd the nonymous referees for their helpful comments. We lso wish to thnk Adm Stubblefield for mny discussions on the nodeid ssignment problem. This work ws supported in prt by grnts from Tes ATP ( ) nd NSF (CCR ). References [1] M. Bellre nd P. Rogwy. The ect security of digitl signtures- How to sign with RSA nd Rbin. In Advnces in Cryptology - EUROCRYPT 9, Lecture Notes in Computer Science, Vol. 17. Springer-Verlg, 199. [2] Steve Bellovin. Security spects of Npster nd Gnutell. In 21 Useni Annul Technicl Conference, Boston, Msschusetts, June 21. Invited tlk. [3] Miguel Cstro, Peter Druschel, Y. Chrlie Hu, nd Antony Rowstron. Eploiting network proimity in peer-to-peer overly networks. Technicl Report MSR-TR-22-82, Microsoft Reserch, My 22. [4] Miguel Cstro nd Brbr Liskov. Prcticl byzntine fult tolernce. In Proceedings of the Third Symposium on Operting Systems Design nd Implementtion (OSDI 99), New Orlens, Louisin, Februry [] In Clrke, Oskr Sndberg, Brndon Wiley, nd Theodore W. Hong. Freenet: A distributed nonymous informtion storge nd retrievl system. In Workshop on Design Issues in Anonymity nd Unobservbility, pges , July 2. ICSI, Berkeley, Cliforni. [] Thoms H. Cormen, Chrles E. Leiserson, nd Ronld L. Rivest. Introduction to Algorithms. MIT Electricl Engineering nd Computer Science Series. MIT Press, 199. [7] Frnk Dbek, M. Frns Kshoek, Dvid Krger, Robert Morris, nd Ion Stoic. Wide-re coopertive storge with CFS. In Proc. ACM SOSP 1, Bnff, Cnd, October 21. [8] Drew Den nd Adm Stubblefield. Using client puzzles to protect TLS. In 1th Useni Security Symposium, pges 1 8, Wshington, D.C., August 21. [9] Roger Dingledine, Michel J. Freedmn, nd Dvid Molnr. Accountbility mesures for peer-to-peer systems. In Peer-to-Peer: Hrnessing the Power of Disruptive Technologies. O Reilly nd Assocites, November 2. [1] John R. Douceur. The Sybil ttck. In Proceedings for the 1st Interntionl Workshop on Peer-to-Peer Systems (IPTPS 2), Cmbridge, Msschusetts, Mrch 22. [11] M. P. Herlihy nd J. M. Wing. Aioms for Concurrent Objects. In Proceedings of 14th ACM Symposium on Principles of Progrmming Lnguges, pges 13 2, Jnury [12] A. Herzberg, M. Jkobsson, S. Jrecki, H. Krwczyk, nd M. Yung. Proctive public key nd signture systems. In Proc. of the 1997 ACM Conference on Computers nd Communiction Security, [13] Ari Juels nd John Brinrd. Client puzzles: A cryptogrphic defense ginst connection depletion ttcks. In Internet Society Symposium on Network nd Distributed System Security (NDSS 99), pges 11 1, Sn Diego, Cliforni, Februry [14] Nncy Lynch, Dhli Mlkhi, nd Dvid Rtjczk. Atomic dt ccess in content ddressble networks. In Proceedings for the 1st Interntionl Workshop on Peer-to-Peer Systems (IPTPS 2), Cmbridge, Msschusetts, Mrch 22. [1] Rlph C. Merkle. Secure communictions over insecure chnnels. Communictions of the ACM, 21(4): , April [1] Sylvi Rtnsmy, Pul Frncis, Mrk Hndley, Richrd Krp, nd Scott Shenker. A sclble content-ddressble network. In Proc. ACM SIGCOMM 1, Sn Diego, Cliforni, August 21.

16 [17] Antony Rowstron nd Peter Druschel. Pstry: Sclble, distributed object loction nd routing for lrge-scle peer-to-peer systems. In Proc. IFIP/ACM Middlewre 21, Heidelberg, Germny, November 21. [18] Antony Rowstron nd Peter Druschel. Storge mngement nd cching in PAST, lrge-scle, persistent peer-to-peer storge utility. In Proc. ACM SOSP 1, Bnff, Cnd, October 21. [19] Emil Sit nd Robert Morris. Security considertions for peerto-peer distributed hsh tbles. In Proceedings for the 1st Interntionl Workshop on Peer-to-Peer Systems (IPTPS 2), Cmbridge, Msschusetts, Mrch 22. [2] Ion Stoic, Robert Morris, Dvid Krger, M. Frns Kshoek, nd Hri Blkrishnn. Chord: A sclble peer-to-peer lookup service for Internet pplictions. In Proc. ACM SIGCOMM 1, Sn Diego, Cliforni, August 21. [21] Ben Y. Zho, John D. Kubitowicz, nd Anthony D. Joseph. Tpestry: An infrstructure for fult-resilient wide-re loction nd routing. Technicl Report UCB//CSD , U. C. Berkeley, April 21. Appendi This ppendi describes n nlytic model for the probbility of flse positives nd negtives in the routing filure test. We ssume tht there eist N nodeids distributed uniformly t rndom on n intervl of length D = If N is lrge nd we look t the K nodeids closest to n rbitrrily chosen loction on this intervl (for some K N), the loction of these K nodeids is well pproimted in distribution by Poisson process of rte N/D. In prticulr, the inter-point distnces re pproimtely independent eponentil rndom vribles with men D/N. Let F 1 denote the eponentil distribution with men µ 1 = D/N nd F 2 the eponentil distribution with men µ 2 = D/N f, where f is the frction of fulty nodes. Suppose y 1,...,y k re independent identiclly distributed (iid) nd re drwn from one of these two distributions nd we re required to identify which distribution they re drwn from, e.g., y 1,...,y k cn be prospective set of replic roots in Pstry nd we re trying to determine if the set is correct or if it contins only fulty nodes. An optiml hypothesis test is bsed on compring the likelihood rtio to threshold; by writing down the likelihood rtio, we see tht this is equivlent to compring the smple men, denoted µ y, to threshold T. We re in sitution where N is unknown but we hve smples 1,..., n from F 1 (i.e., the smples tht we collect from the nodeids close to the sender in the id spce). We propose the following hypothesis test: choose threshold of the form γµ, for some constnt γ (1, 1/ f ), nd ccept/reject the hypothesis tht Y i re iid F 1 by compring µ y to this threshold. We now compute the flse positive probbility, α, nd the flse negtive probbility, β, for this test. Denote n/k by r nd ssume without loss of generlity tht r is n integer. For i = 1,...,k, define Z i = Y i γ r (X (i 1)r X ir ), nd note tht the Z i re iid rndom vribles. Let S j denote the sum of j iid eponentil rndom vribles with men µ 1 = D/N. The event tht µ Y > γµ X is then the event tht k i=1 Z i >. Thus, k α(n,k,γ) = P 1 ( Z i > ) = P( 1 i=1 k S k > γ n S n), (1) where we write P 1 to denote probbilities when the Y i hve distribution F 1. Reclling tht S j hs the gmm distribution with shpe prmeter j nd scle prmeter 1/µ 1, we cn rewrite the bove s (/µ α(n,k,γ) = 1 ) n 1 µ 1 (n 1)! e µ 1 (/µ 1 ) k 1 γk µ n 1 (k 1)! e µ 1y dyd = nn k k e n k u n 1 e n(u 1) v k 1 e k(v 1) dvdu (n 1)!(k 1)! (n 1)! γu (k 1)! where we used the chnge of vribles u = /(nµ 1 ) nd v = y/(kµ 1 ) to obtin the lst equlity. This epression cn be used to compute α numericlly. We now derive simple closed-form epression for n upper bound on α. The bound shows tht α decys eponentilly in the smple size, k, nd in fct cptures the ect eponentil rte of decy. For rbitrry θ, we hve by Chernoff s bound tht k ( ) ( k α E[ep(θ Z i )] = E[e θy 1 ] E[ep( γθ ) rk i=1 r X 1)] Now, if X hs n eponentil distribution with men µ, then E[e θx ] is 1/(1 θµ) for θ < 1/µ nd + for θ 1/µ. Thus, for ll θ [,1/µ 1 ), we hve logα k log(1 θµ 1 ) rk log(1 + γθµ 1 ) r The tightest upper bound is obtined by minimising the epression on the right over θ [,1/µ 1 ). The minimum is ttined t θ = r+1 r γ 1 γµ 1. Substituting this bove yields the bound, { [ α ep k (r + 1)log r + γ ]} logγ (2) r + 1 We cn derive n epression for the flse negtive probbility, β, long similr lines. Now, the Y i re iid with distribution F 2, i.e., they re eponentilly distributed with men µ 2 = µ 1 / f, nd we re interested in the event tht µ Y γµ X. If this hppens, then we fil to reject the hypothesis tht the Y i hve distribution F 1. Thus k β(n,k,γ, f ) = P 2 ( Z i ), i=1 where we write P 2 to denote probbilities when the Y i re eponentil with men µ 1 / f. In this cse, Y 1 hs the sme distribution s X 1 / f, so k i=1 Y i hs the sme distribution s ( k i=1 X i)/ f, nd we obtin using (1) tht β(n,k,γ, f ) = P( 1 k S 1 k f < γ n S2 n ) = P( 1 n S2 n > 1 1 γ f k S1 k ) = α(k,n, 1 γ f ) This llows us to compute β numericlly nd by combining this with (2), we obtin the following closed-form upper bound { [ β ep k (r + 1)log r + γ f ]} log(γ f ) r + 1