Similarity Evaluation Scheme Using Fixed-length VLC Based Representative Hash

Transcription

1 Similarity Evaluation Scheme Using Fixed-length VLC Based Representative Hash Young Jun Yoo 1, Jin Kim, Jung min So, Jeong Gun Lee, Sun Jung Kim,Young Woong Ko 1 Dept. of Computer Engineering, Hallym University Chuncheon, Korea { willow72, jinkim, jso, jeonggun.lee, sunkim, yuko}@hallym.ac.kr Abstract. File similarity evaluation is essential for data deduplication and similar file searching. In the file similarity processing time, the CPU consumption and resource overhead of memory are increased as the number of files increase. Moreover, as the file size is getting bigger, the overhead of metadata capacity is critical. In this paper, we suggest the similarity evaluation scheme using Fixed-length VLC with representative hash scheme, which reduce overall processing time of similarity evaluation. Experiment result shows that the proposed system can reduce processing time and data bandwidth. Keywords: File, Synchronization, chunking, Hash, FLC 1 Introduction File similarity evaluation is used in various fields such as malicious file detection and data deduplication. In security area, malicious files are generated by modifying original executable file, so the file similarity search is used to identify file similarity or find malicious code from the executable programs within the system. Usually, the malicious file is very similar with original file because file patch is appended in the middle of original file. Data deduplication system also handles similar files with various chunking approach. The well-known chunking approaches are FLC(Fixedlength chunking) and VLC(Variable-length chunking). Deduplication approaches can save computing resources through detecting the file similarity and eliminating the duplicated region of a file. In file similarity searching system module, the performance depends on hash comparison speed, therefore effective hash comparison is very critical. In this paper, we propose file similarity evaluation system which determines similarity between client files and the server files. By searching high similarity files, we can reduce file synchronization overhead. This technique is much faster than traditional file synchronization systems which compare hash data of a client to the 1 This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education, Science and Technology(.2012R1A1A ) and this research was supported by the MKE, Korea and NHN. under IT/SW Creative research program supervised by the NIPA (NIPA-2012) IST 2013, ASTL Vol. 23, pp , 2013 SERSC

2 Proceedings, The 2nd International Conference on Information Science and Technology server one. The proposed scheme adapts VLC approach for data chunking, however it only hashes fixed-length data block for extract hash values. Also the proposed scheme reduces hash comparison time by using representative hash scheme which generates a hash list that only contains key feature hash list of a file. 2 Related works Currently, there are several well-known file synchronization schemes including Rsync[1], Venti[2], Active Sync[3], HotSync[4] and CPISync[5]. Those approaches lacks of checking file similarity, so it is difficult to apply those scheme to versioning system where frequently data is modified and has lots of data duplication between files. There are more elaborated research results where each scheme considers file duplication and provides efficient file synchronization; LLRFS[6] use CDC(Contents Define Chunking) and set reconciliation to diminish data duplication and minimize network traffic. Tpsync[7] finds duplicated region in a file with CDC and apply rolling checksum for more enhanced deduplication. Rsync[1] is a software application for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of Rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction. Rsync can copy or display directory contents and copy files, optionally using compression and recursion. Venti[2] is a network storage system that permanently stores data blocks. A 160-bit SHA-1 hash of the data(called score by Venti) acts as the address of the data. This enforces a write-once policy since no other data block can be found with the same address. The addresses of multiple writes of the same data are identical. So duplicate data is easily identified and the data block is stored only once. 3 Design and Implementation of the system In this paper, we designed and implemented the file similarity evaluation scheme for file synchronization system. In file synchronization is the process of ensuring that files in multiple locations are updated via certain rules. Updated files are copied from a source location to one or more target locations, or updated files are copied in both directions. The purpose of file synchronization is to keep the file of multiple locations identical to each other. The file synchronization is widely used for backups or updating on storage systems and it is especially useful for mobile devices, or others that work on multiple computers. To reduce data bandwidth, file synchronization has to prevent copying already identical files and thus have to save processing time during file copying. In this paper, we propose a practical approach for minimizing metadata communication overhead for remote file synchronization system. The main idea of this paper is to apply efficient hash comparison technique by reducing hash keys of file into small number of hash keys using representative hash selection. We designed and implemented the proposed file synchronization scheme on the static chunking 305

3 Similarity Evaluation Scheme Using Fixed-length VLC Based Representative Hash deduplication system. The chunking approaches are often used as separated to FLC (Fixed-Length Chunking) and VLC (Variable-Length Chunking) to find file similarity. VLC operation module is much higher effective to determining file similarity than other approach but high probability to big overhead of comparing process. Fixedlength chunking approach is dividing file into block by predefined constant length and calculating hash (MD5, SHA1 ) values for each block. Then accumulates hashes to hash-list and comparing these values to file hashes in the server to finds file similarity. FLC performs very fast because file is divided into blocks by constant length. The proposed system reads whole file stream, chunking into fixed-size blocks starting from each anchor position and generates hash values. The hash values are reduced by using the representative hash scheme. Figure 1. Pseudo code of proposed system The algorithm (figure 1) shows how the proposed system performing chunking and hashing. First, the algorithm searches anchor bytes in a file stream. Second, if it finds anchor bytes in a file stream, it extracts block hash from the fixed-sized block starting from the anchor. Figure 2. Representative hash concept 306

4 Proceedings, The 2nd International Conference on Information Science and Technology As can be seen in figure 2, Rabin hash function is used for computing a hash key for a block. The Rabin hash starts at each byte in the first byte of a file and over the block size of bytes to its right. If the Rabin computation at the first byte is completed then we have to compute the Rabin hash at the second byte incrementally from the first hash value. Now that the hash value at the second byte is available then we use it to incrementally compute the hash value at the third, and continue this process. We have to sort the Rabin hash value and choose only 10 maximum values as a representative hash. In this work, we made the representative hash list for all files before data deduplication. We extract one representative hash for 1 MByte therefore the amount of additional information for file similarity is not critical for metadata management. 4 Experiment Result We now present some experimental results that show the potential of the proposed algorithm. To perform comprehensive analysis on the proposed algorithm, we implemented the client and the server on the platform that consist of 3GHz Pentium 4 Processor, WD-1600JS hard disk and 100Mbps network. We made experimental data set using patch that means data block used for modifying a file in a random manner. In this experiment, we modified a file using lseek function in Linux system using random file offset and applied a patch to make test data file. For each run, we did multiple runs with different data sets, and plot the average resulting value. Experiment original data is modified as 90%, 80% and 70% duplicates by Linux dd command. Experiment goal is to measure the evaluation result of file similarity by Fixed-size VLC with representative hash(fvr) and traditional FLC approaches. Also we also measured error size when we measured file similarity. Table-1. File similarity experiment result FLC FVR Number of hash Perform speed (sec) Table-1 shows the performance result of execution speed. To see experiment result more detail, FLC file similarity approach produces hashes and FVR approach only produces 500 representative hashes when performing the hash operation to the same file. The difference between performing speed of traditional FLC in 0.4 sec and FVR in sec. Table-3. Performing error distribution Actual file similarity Perform result(%) Error(%) 90(%) (%) (%)

5 Similarity Evaluation Scheme Using Fixed-length VLC Based Representative Hash Table-2 shows errors of the proposed system performing result. From the experiment proposed system can determine each 88,4% duplicates of real 90 duplicates which 1.6% loss 81% of 80 real duplicates which 1% loss and 73.8% duplicates of 70% real duplicates which 3.8% loss in experiment. Therefore proposed system is considered to be proved to can determine file similarity in very high rate with fast time. 5 Conclusion In this paper, we introduced structure for representative hash based file similarity evaluation technique which almost similar results as traditional FLC but reducing overhead. By searching high similarity files, we can reduce file synchronization overhead. This technique is much faster than traditional file synchronization systems which compare hash data of a client to the server one. The proposed scheme adapts VLC approach for data chunking, however it only hashes fixed-length data block for extract hash values. References 1. Tridgell, A.: Efficient algorithms for sorting and synchronization. PhD thesis, The Australian National University (1999) 2. Quinlan, S., Dorward, S.: Venti: a new approach to archival storage. In: Proceedings of the FAST 2002 Conference on File and Storage Technologies. Volume 4 (2002) 3. Meunier, P., Nystrom, S., Kamara, S., Yost, S., Alexander, K., Noland, D., Crane, J.: Activesync, tcp/ip and b wireless vulnerabilities of wince-based pdas. WETICE 2002, IEEE International workshops on (2002) 4. HotSync, P.: Palm developer online documentation (2007) 5. Starobinski, D., Trachtenberg, A., Agarwal, S.: Efficient pda synchronization. Mobile Computing, IEEE Transactions on 2(1) (2003) 6. Yan, H., Irmak, U., Suel, T.: Algorithms for low-latency remote synchronization. In: INFOCOM The 27th Conference on Computer Communications. IEEE (2008) 7. Xu, D., Sheng, Y., Ju, D., Wu, J., Wang, D.: High effective two-round remote file fast synchronization algorithm. 5(1) (2011) 308