GlusterFS: Transparent encryption in distributed systems with non-trusted servers Edward Shishkin edward@redhat.com
Agenda: The new GlusterFS functionality: transparent encryption in the systems with non-trusted servers. The encryption/crypt translator. We'll start from the common tasks that have been raised. Then we'll consider encountered problems and their solutions, that have determined the design.
Trusted and non-trusted machines Suppose you are a user of this feature (transparent encryption and authentication in GlusterFS). You qualify every machine as either trusted, or non-trusted. Examples: 1. Your personal laptop, which is under your supervision is trusted machine. 2. Remote GlusterFS servers, which are not under your supervision are non-trusted machines. They are managed by a good admin, but you don't trust him your private data. 3. Clouds are important example of a set of non-trusted machines: you don't know what is going on there at all.
Trusted and non-trusted objects Every machine contains objects (in the RAM, disks, registers, etc). All objects of every non-trusted machine are non-trusted by definition. Trusted machine contains both type of objects (trusted and non-trusted). Sources of non-trusted objects on your trusted machine: non-trusted media; non-trusted network; social engineering; etc. Sources of trusted objects on your trusted machine: trusted media; trusted network; process of verification of non-trusted objects (authentication).
Examples of trusted and non-trusted objects 1.Your secret key properly generated on your trusted machine, or retrieved from trusted media is trusted object. 2.You wanted to look at user accounts on your trusted local machine. The string "/etc/passwd" that you have passed to the open(2) is trusted object. 3.Someone you don't know asked you to check if you have a file "/foo/bar" on your trusted local machine. The string "/foo/bar", that you have passed to readdir(2) is non-trusted object. 4.Encrypted content of any regular file received from the non-trusted GlusterFS servers before processing by the crypt translator is non-trusted object. 5.Decrypted content of your regular file after successful processing by the crypt translator is trusted object. 6.List of file names provided by ls(1) for your mounted encrypted GlusterFS volume is nontrusted object (see subsection 1 of the Restrictions below).
The common tasks U S E R plain text cipher transform crypt xlator cipher text Trusted client N E T W O R K cipher text disk Non-trusted server (data and metadata can be easily subjected to tampering and analysis by unauthorized parties) 1) Encrypt user data on the client side, store encrypted data on the server; 2) Detect any silent tampering, which doesn't look like data corruption, but leads to revealing private user's data 3) Make sure everything is POSIX and NIST compliant
Restrictions 1.We encrypt only file content. The feature of transparent encryption doesn't protect file names: they are neither encrypted, nor verified. Protection of file names is not so critical as protection of encryption-specific file's metadata: any attacks based on tampering file names will break POSIX compliance and result in massive corruption, which is easy to detect. 2.The feature of transparent encryption is not supported in NFS mounts of GlusterFS volumes: NFS's file handles introduce security issues, which are hard to resolve. 3.The feature of transparent encryption is incompatible with GlusterFS performance translators (quick-read, write-behind and open-behind).
Properties of cipher algorithms, that have determined the design Property 1. Atomicity Algorithm introduces dependencies between bytes of ciphertext / plaintext: plain text x 1 x 2... x k y j =f (x 1, x 2,..., x k ) cipher text y j x i =g( y 1, y 2,..., y k ) To calculate a byte of cipher [plain] text we Need to know all bytes of the whole chunk of plain [cipher] text. This chunk is called the enveloping atom
Resolving FOP->read() and FOP->write() off cnt 0 1 2 3 4 off' cnt' Enveloping set of atoms (off, cnt) -> (off', cnt'): translate user's chunk (red) to the enveloping set (blue) of atoms (#1, #2, #3) read(off, cnt, buf) -> read(off', cnt', buf'); decrypt(buf'). write(off, cnt, buf) -> read(off', cnt', buf'); decrypt(buf'); modify(buf'); encrypt(buf'); write(off', cnt', buf').
Problem #1: RMW of local buffers leads to non-posix behavior. Example: Process #1: Process #2: Read first 8 bytes of file A to the local buffer of the #1 Modify byte #0 Write the modified buffer Read first 8 bytes of file A to the local buffer of the #2 Modify byte #7 Write the modified buffer Modification performed by the process #2 is lost!
Property 2. EOF issue Cipher algorithm requires input stream size to be a multiple of cipher block size (and produce output stream of such size). Problem #2 In addition to standard file attributes, we need to keep information about actual size of encrypted file.
Problem #3: Holes in encrypted files write(off,cnt), off > file_size off 0 file_size cnt this portion of data is represented in local file system with special signatures, which indicate a hole This leads to non-posix behavior (users should observe a hole as a set of zeros). We should prevent creating such signatures in the host local file system. And the most reasonable way is to convert the hole to zeros before the hole punching operation. To convert hole properly we need to know file size and to make sure that it won't be changed by a concurrent process.
To resolve problems #1, #2 #3 we need to acquire an exclusive access to the file (or to a part of the file): For #1: to make sure before performing a read- modify-write sequence that our local buffer won't be obsoleted by a concurrent process. For #2, #3: to make sure that file size won't be changed by concurrent process during EOF handling (during hole conversion).
Handling EOF (End Of Files) Before encryption file's tail is padded with zeroes; Encrypted file body of padded size is stored on the server; Actual (non-padded) file size is stored as xattr of this file; We introduce a special translation layer in the crypt translator: every time when any callback returns file attributes, we update ia_size with actual ( non-padded ) value
Implementation of FOP->ftruncate(): acquire an exclusive access; extract actual file size; If this is expanding truncate, then:. write zeros; else (shrinking truncate):. read the partially pruned atom (#2);. perform the cut-pad-encrypt sequence;. write the result; 0 write 1 2 3 pad new size new_size with padding old_size
Implementation of FOP->truncate() crypt_open(); /* open the file */ crypt_ftruncate(); /* described above */ crypt_flush(); /* release */
Generating keys for data encryption Every file is encrypted with its own unique key (NIST recommendation for XTS cipher mode), which is calculated at ->open() time by the following way: K_fd = f(k_vm, object_id); K_fd per-file key for data encryption; K_vm per-volume master key; object_id trusted unique file id (GFID); f NIST compliant derivation function.
Authentication requirements Data: file data authentication is recommended, but not a must, as we encrypt / decrypt data with XTS cipher mode. Tampering a ciphertext created in XTS mode leads to unpredictable changes inplain text. Metadata: Some metadata (like GFID) is used to derive cipher keys, and other private info. Authentication of such critical metadata is a must.
Authentication requirements (continue) We perform metadata authentication by creating/checking MACs. Both, metadata and MACs are initially created on the trusted client machine. That is, we should: 1) create/check MAC of the metadata to make sure that metadata is not tampered); 2) check the ownership (make sure that metadata belongs to proper file).
The requirement (2) above is essential! Indeed, let's consider verification without checking the ownership relations. Creating a MAC at FOP->create() time: MAC = F(mtd); mtd metadata string; mtd MAC file body Verifying the MAC at FOP->open() time: if (MAC == F(mtd)) return verification_ok; return verification_failed;
Example of tampering: breaking ownership relations between files and pairs mtd-mac mtd1 MAC1 file1 (empty) before tampering mtd2 MAC2 data2 file2 mtd2 MAC2 file1(empty) after tampering mtd2 MAC2 data1 file1(non-empty) after tampering After substitution (mtd1, MAC1) with (mtd2, MAC2) metadata verification will be OK. However, data of files file1 and file2 will be encrypted with the same key. There is a known class of attacks based on such key scope extensions.
Metadata verification with checking ownership relations MAC MAC = F(mtd, F(mtd, trusted_file_id) trusted_file_id) mtd metadata string; trusted_file_id unique trusted file id. We use the approach, where trusted_file_id is the name of the file. We rely on the fact that it is hard to infect the trusted client machine by some tricky way, so that client will be forced to pass broken file names to the server. Problem : file can have many different names (hard links)
Creating per-link MACs mtd MAC1 MAC2... MACn File with n links Metadata verification and checking ownership relations by hardlink names MAC1 = F(mtd, link_name_1); MAC2 = F(mtd, link_name_2);... MACn = F(mtd, link_name_n); for (i=0; i < n; i++) if (MACi == F(mtd, name)) return verification_ok; return verification_failed;
Metadata verification with checking ownership relations by hardlink names: Proof of correctness mtd1 MAC11 MAC12... MAC1n File1, n links MAC11 = F(mtd1, link_name_11); MAC12 = F(mtd1, link_name_12);... MAC1n = F(mtd1, link_name_1n); File2, k links mtd2 MAC21... MAC2k MAC21 = F(mtd2, link_name_21); MAC22 = F(mtd2, link_name_22);... MAC2k = F(mtd2, link_name_2k); Substitution of (mtd1, MAC1p) with (mtd2, MAC2q) will lead to failed verification, as MAC2q!= F(mtd2, link_name_1p), because link_name_1p!= link_name_2q for any p = 1,..., n; q = 1,..., k;
Metadata string This is a per-file object, which represents encoded attributes needed for data encryption and metadata authentication. Created by create_format at FOP >create() time. Stored as xattr of the file with magic name defined by macro CRYPTO_FORMAT_PREFIX Format: version number (1 byte) Version-dependent body Updated by: FOP >link(): insert a new MAC for the new name; FOP >rename(): update the respective MAC; FOP >unlink(): cut the respective MAC; Verified and extracted by FOP >open().
Format V1 of metadata string (id number 0), struct mtd_format_v1 field oid description Unique object_id (GFID, 16 bytes) NMTD, Not encrypted attributes field alg_id mode_id block_bits minor_id dkey_factor description Data cipher algorithn id (1 byte) Data cipher mode id (1 byte) Encoded block size (1 byte) encryption translator id (0 for crypt) Encoded size of data cipher key EMTD, encrypted attributes field gmac omac MAC of EMTD description Array of per-link MACs of NMTD MACs
Hierarchy of keys K_vm K_vn K_fe K_fd K_ln K_vm: per-volume master key. Provided by user at mount time. K_vn: per-volume nmtd key. Its material is used to generate K_ln; K_fe: per-file key for EMTD encryption and authentication; K_fd: per-file key for data encryption; K_ln: per-link key for NMTD authentication.