A Decentralized Method for Data Replication Among Computing Nodes in a Network

Transcription

1 A Decentralized Method for Data Replication Among Computing Nodes in a Network Lars K. Holte January 25, 2015 Abstract A method is presented whereby computing nodes can send data to their peers on a network for the purpose of replication ( backup ). The total number of copies of a given piece of information on the network can be verified by any of the nodes and adjusted by the information s owner. Data can be encrypted using a symmetric key system which prevents nodes from interpreting information they do not own. 1 Introdution No hardware is 100% reliable. This includes hardware designed for persistent storage of information (e.g., computer hard drives). Methods must therefore exist by which important data can be replicated among multiple storage devices. This sort of replication reduces the possibility of permanent data loss to the possibility of simultaneous failure of all devices on which the data is replicated. As long as the devices are not in the same physical location, this chance is fairly small. A fairly common way to implement data replication on a network is to have a single node (a backup server ) which duplicates important information from all the other nodes. This method has three notable limitations: 1. The client nodes must trust that the backup server actually stores the information sent to it, and that the backup server never modifies the information before sending it back. 2. The backup server most often resides in a single location, and the information it stores is therefore susceptible to catastrophic failure from hazards such as fire and water damage. 1

2 3. There are at most two copies of any given piece of information: one on the client node and one on the backup server. The probabilistic benefits of replication do not stop there, however: simultaneous failure of three or four storage devices is far less probable. The decentralized, trustless protocol for data replication detailed herein exhibits none of these characteristics. 2 Definition of Terms These terms clarify the terminology used later and also serve to set up an example case. 2.1 Network Suppose the network consists of N computing nodes. Each node is individually addressable by any other node on the network. There need not be broadcast or multicast addresses. It is assumed that failures resulting in loss of stored data on each node are probabilistically independent. 2.2 Node Each node has storage space, network connectivity, and computing power. The storage space on the node can be understood to contain four different types of data: 1. The master copies of important pieces of information which were originally created on this node and must be replicated on other nodes of the network. 2. Important information from other nodes on the network. 3. Unused space. 4. Buffer space so that the filesystem does not reach 100% capacity in a worst-case scenario. Each node has a public and private key, possibly separate from the public and private key the node may use to communicate securely over the network. 2

3 2.3 Replication Group A replication group (r-group) is a set of nodes within the network which have agreed to share the burden of replicating a certain set of data. Each node may belong to zero or more replication groups. Each replication group has a characteristic replication ratio, r, a positive integer greater than or equal to 1, which defines how many copies of each piece of information should exist among all the nodes of the group at any given time. Each replication group is identified by a UUID. Some groups might be open, meaning that any node can join simply by asking. Some groups might be closed, meaning that a password is required to join. Closed groups could also have dedicated sentry node(s), maintained by an administrator, which could be configured only to allow certain nodes (identified by their public keys) to enter. 2.4 Fragment A fragment is a piece of information which needs replication. Each fragment is owned by the node at which it originated. Each fragment is identified by its owner s public key and a 64-bit sequence number assigned to it by the owner. For all practical purposes, it may be assumed that the first 64 hexadecimal digits of the public keys of all nodes on the network are distinct. When storing a fragment for another node, therefore, the information may be identified by the 64 hexadecimal digits and the 64-bit number. This allows nodes within an r-group to easily store fragments from other nodes in a two-level directory structure. All information which needs to be replicated must be divided into fragments. For instance, if a node wishes to push an entire file out to an r-group, it must divide that file up into fragments. Only the owner maintains the knowledge of how fragment numbers are assigned, and which ones must come together to reconstruct any given piece of information. Fragments may have any size in bytes from 1 byte to 64KiB (this allows the size of each fragment to be stored as a 16-bit unsigned integer). 3 Processes 3.1 Joining an R-Group Suppose a node wants to join a specific replication group, and has the UUID for that group. It will contact one of the nodes on this group (knowing the address of at least one node in the group is a prerequisite) and ask 3

4 to join. What happens next depends on the group type. For an open group, the queried node will return a list of all the other nodes in the group. The interested node should then transmit a message to each of the nodes in the list indicating that it has joined the replication group. It is not necessary that the list be complete. For a closed group, the sentry node may prompt for a password, as discussed. Joining a group may also require human intervention, in the form of completing a Captcha, for instance. 3.2 Submitting Information for Replication Suppose important information is created on a node. A file, for instance. This node (the owner ) must divide the information into fragments, and submit each of these fragments to the other nodes in a replication group. Any reversible algorithm may be used to divide the file into fragments. Before each fragment is submitted, it is encrypted using a reversible, cryptographically secure cipher such as AES256. The cipher key is based on the node s private key. In this way the peer nodes cannot interpret the data. Each peer may accept or refuse the fragment. It may do this, for instance, based on the amount of space it has remaining. The client node continues until at least r nodes have accepted each fragment. A few minutes later, the client polls the accepting nodes to see how many can still provide the fragment. If any are unable, the client again searches the r-group for acceptors until the number of replications is r. The client node periodically performs this check. How frequently this periodic checking occurs depends on the level of trust in the network. 3.3 Determining Whether a Peer Has a Fragment In a trusted r-group, a simple query/response would be sufficient. If the node is not trusted, the original fragment must be returned verbatim. A checksum would be sufficient to prove that the peer has actually stored the data, but this opens up an attack avenue: an untrusted peer may refuse to return the actual data, returning only checksums. 3.4 Deleting a Fragment Suppose an important piece of information is deleted from the node on which it originated. The owning node must communicate to its peers that it is no longer necessary to replicate the fragments associated with that information. To do this, it sends out signed deletion messages to each node which it has recorded as storing each fragment. The nodes delete these fragments upon 4

5 receiving the message. Should a node lose its private key, it will be unable to sign deletion messages, and that information will remain resident in the replication group. No node will be able to interpret it, because the owner has lost its key. Over time, if a node needs space, it may offload some of the least-requested fragments from other nodes. In this way the information from an inactive node eventually exits the r-group. 4 Attacks The network has mechanisms for preventing untrustworthy nodes from interpreting data they do not own, from using too much disk space in an r-group, and from deleting information from the network. 4.1 Unauthorized Deletion Problem: A rogue node sends out messages indicating that a fragment it does not own is to be deleted, in an attempt to erase that information from the network. Solution: Deletion messages must be signed by the private key of the owner. It is only possible to delete another node s fragments if the private key of the owner is compromised. Additionally, without detailed knowledge of a node s state, it will be impossible for an attacker to tell which fragment numbers it has used, which are unused, and which (other) nodes in a given r-group hold copies of those fragments. Total deletion, even with a compromised key, would require a brute-force attack in two spaces: fragment address space (2 64 ) and r-group space (the number of nodes in the r-group). 4.2 Unauthorized Access Problem: A rogue node attempts to obtain fragments owned by another node. Solution: The owner s signature is required on any message querying the status of a fragment or resulting in the transmission of the fragment. 4.3 Unauthorized Interpretation Problem: A rogue node attempts to interpret information which it stores but does not own. 5

6 Solution: Each fragment passes through a cryptographically secure cipher before reaching other nodes on the network, so it is practically infeasible to recover the information without the cipher key. 4.4 Data Spamming Problem: A rogue node generates many junk fragments in an attempt to use up disk space in an r-group. Solution: It is important to note that this is identical to the situation in which a trustworthy node produces reams of useful information: the only difference is in the interpretation of the data. The best approach is to limit the amount of information any given node in an r-group will store for any single network address or public key identifier. This limits the total amount of information that any node can store on the network. Within a private ( closed ) r-group, such limits could be adjusted for certain nodes. 4.5 Creation of False Nodes Problem: A rogue node creates many asymmetric key pairs (node identifiers) and acts on behalf of many nodes, attempting to bypass the storage limit allocated for each node in an r-group. Solution: As mentioned above, storage limits are based on a matching network address or public key identifier. The storage limit cannot be bypassed, therefore, unless the node is also able to change its address. 4.6 Creation of False Nodes at Multiple Addresses Problem: A rogue node creates an asymmetric key pair (node identifier), joins an r-group, and then spams it until most fragments are refused by its peers, indicating that it has met its storage limit. The node then changes its address (for instance, through DHCP), generates a new asymmetric key pair, and repeats the process. Solution: There is no generally-applicable solution. There are three things to note: 1. Dynamically changing your address may not be possible on a public network. 2. In most cases, creation of a node will imply human intervention. It would not be unreasonable, therefore, to require a node to complete 6

7 a Captcha or some other such task best performed by a human before allowing it entrance to an r-group. This would make the attack considerably less convenient. 3. Unused fragments will be offloaded first, meaning that a lot of network activity will be required for the rogue node to maintain a large share of the total disk space if the other nodes are requesting smaller amounts and total disk space is scarce. 5 Conclusion This document has outlined in broad strokes a protocol by which computing nodes on a network can share storage space for the purpose of replicating information. The nodes do not have to trust each other, and important information can be replicated more than twice. Fairly few specifics have been given about the implementation of this protocol, but I hope that the description has been clear enough to convince the reader its implementation would not be impossible. 7