Chapter 8: On the Use of Hash Functions in. Computer Forensics

Transcription

1 Harald Baier Hash Functions in Forensics / WS 2011/2012 2/41 Chapter 8: On the Use of Hash Functions in Computer Forensics Harald Baier Hochschule Darmstadt, CASED WS 2011/2012 Harald Baier Hash Functions in Forensics / WS 2011/2012 1/41 Fuzzy Hashing

2 Harald Baier Hash Functions in Forensics / WS 2011/2012 4/41 Fuzzy Hashing Harald Baier Hash Functions in Forensics / WS 2011/2012 3/41 Use Case: Prosecution 1. Police and prosecutors confronted with different storage media: Hard disk drives, solid-state drives, USB sticks. Mobile phones, SIM cards. Digital cameras, digital camcorders, SD cards. CDs, DVDs. RAM (dumps) Amount of distrained data often exceeds 1 terabyte.

3 Harald Baier Hash Functions in Forensics / WS 2011/2012 6/41 Different views of 1 terabyte 1 terabyte of digital text is (approximately) equal to: 1. 1 trillion characters: 1 character = 1 byte million pages: 1 page = 5000 characters years of printing time: 20 sheets per minute million kg of paper: onesided printed. 5. Paper stack of 22 km height: bulk of 0.1 mm. Harald Baier Hash Functions in Forensics / WS 2011/2012 5/41 Finding relevant files resembles... Source: tu-harburg.de Source: beepworld.de

4 Harald Baier Hash Functions in Forensics / WS 2011/2012 8/41... or is it solved for suspect files? Harald Baier Hash Functions in Forensics / WS 2011/2012 7/41 Fuzzy Hashing

5 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Definition and Security Applications 1. A hash function h is a function with two properties: Compression: h : {0, 1} {0, 1} n. Ease of computation: Computation of h(m) is fast in practice. 2. Notation: m is a document (e.g. a file, a device). h(m) its hash value or digest. 3. Sample security applications: Storage of passwords. Electronic signatures (MAC, asymmetric signatures). Computer forensics. Harald Baier Hash Functions in Forensics / WS 2011/2012 9/41 Hash Functions in Cryptography For use in cryptography, we have to impose further conditions: 1. Preimage Resistance: Let a hash value H {0, 1} n be given. It is infeasible in practice to find an input (i.e. a document m) with H = h(m). 2. Second Preimage Resistance: Let a document m {0, 1} be given. It is infeasible in practice to find a second document m with m m and h(m) = h(m ). 3. Collision Resistance: It is infeasible in practice to find any two documents m, m with m m and h(m) = h(m ).

6 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Avalanche Effect 1. Let m and h(m) be given. If m is replaced by m, h(m ) behaves pseudo randomly. = No control over the output, if the input is changed. 2. Consequences: If only one bit in m is changed to get m, the two outputs h(m) and h(m ) look very different. Every bit in h(m ) changes with probability 50%, independently of the number of different bits in m. Harald Baier Hash Functions in Forensics / WS 2011/ /41 Sample Cryptographic Hash Functions Name MD5 SHA-1 SHA-256 SHA-512 RIPEMD-160 n Demo: 1. Computation of hash values using openssl. 2. Avalanche effect. 3. Performance.

7 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Fuzzy Hashing Harald Baier Hash Functions in Forensics / WS 2011/ /41 Use Cases 1. Ensure authenticity and integrity during data acquisition. Relevant for both dead and live analysis. Hash values must be protected: Written down by hand in investigation notebook. Compute a digital signature over it. 2. Identify known files: Whitelisting: Known to be good files. Blacklisting: Known to be bad files. 3. Bloom filters to efficiently identify a large set of files. Relevant security property of the hash function: Second-preimage resistance.

8 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Data Acquisition: Dead analysis 1. Investigation is performed on a copy. 2. Main goal of hashing: Preserve chain of custody. 3. Common approach for a storage medium: Compute hash value h1 over the whole original volume. Write hash value h1 down in physical logbook. Make a 1-to-1 copy of the volume using dd. Compute hash value h 2 over the copy. Write hash value h2 down in physical logbook. Work read-only on copy. To prove chain of custody, compute h 2 again and check, if h 1 = h 2 holds. Harald Baier Hash Functions in Forensics / WS 2011/ /41 Example: Save first partition of a HDD Chain of custody preserved $ openssl dgst -sha1 /dev/hda1 SHA1(/dev/hda1)= c1e4449bfd504b60e85447afaa3cd100f7e59dad $ dd if=/dev/hda1 of=image-hda1.dd bs=512 $ openssl dgst -sha1 image-hda1.dd SHA1(image-hda1.dd)= c1e4449bfd504b60e85447afaa3cd100f7e59dad [INVESTIGATE read-only image-hda1.dd] $ openssl dgst -sha1 image-hda1.dd SHA1(image-hda1.dd)= c1e4449bfd504b60e85447afaa3cd100f7e59dad

9 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Example: Save first partition of a HDD Chain of custody destroyed $ openssl dgst -sha1 /dev/hda1 SHA1(/dev/hda1)= c1e4449bfd504b60e85447afaa3cd100f7e59dad $ dd if=/dev/hda1 of=image-hda1.dd bs=512 $ openssl dgst -sha1 image-hda1.dd SHA1(image-hda1.dd)= c1e4449bfd504b60e85447afaa3cd100f7e59dad [INVESTIGATE read-only image-hda1.dd] $ openssl dgst -sha1 image-hda1.dd SHA1(image-hda1.dd)= 568c68d97baf f509343b8b02d324e3d8 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Hash Functions in Data Acquisition: Assessment 1. Key problem: If original and copy differ by at least 1 bit, the chain of custody is broken. 2. Solutions: Compute hashes over smaller pieces of the input (e.g. file system blocks). This leads to segment hashes. Perform a binary search over original and copy to find differences.

10 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Whitelisting 1. Underlying Idea: Generate a database of known to be good files. Let G denote this set. Identify automatically an unsuspicious file on base of its hash value, which matches a fingerprint of a file in G. Exclude a known to be good file from further investigation. Significant reduction of irrelevant data. 2. Examples of unsuspicious files: System files of operating systems. Well-known benign applications like browsers, editors, Widespread database: Reference Data Set (RDS) of the National Software Reference Library (NSRL), maintained by NIST Harald Baier Hash Functions in Forensics / WS 2011/ /41 NSRL-RDS 1. Website 2. Current download version: RDS 2.34 from October Original software is kept to ensure court worthiness. 4. Propagated via CD. Download of iso-images from website via http. 5. RDS 2.34 comprises four iso-images of about 1.2 GB size. 6. Contains files and unique SHA-1 values (due to duplications across different software distributions) 7. Entries are arranged according to the SHA-1 values.

11 Harald Baier Hash Functions in Forensics / WS 2011/ /41 NSRL-RDS Sample entries $ less NSRLFile.txt "SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode" " EDD92C4E3D2E F849","392126E756571EBF112CB1C1CDEDF926","EBD105A0","I05002T2.PFB",98865 " DA6391F7F5D2F7FCCF36CEBDA60C6EA02","0E53C14A3E48D94FF596A B492","AA6A7B16","00br2026.gif",2226, "000000A9E47BD385A0A3685AA12C2DB6FD727A20","176308F27DD52890F013A3FD80F92E51","D749B562","femvo523.wav",42748 " AFA836117B1B572FAE4713F200567","9B3702B0E788C6D FE3C9786A","05E566DF","J JPG",32768 " AFA836117B1B572FAE4713F200567","9B3702B0E788C6D FE3C9786A","05E566DF","J JPG",32768 " AFA836117B1B572FAE4713F200567","9B3702B0E788C6D FE3C9786A","05E566DF","J JPG",32768 " AFA836117B1B572FAE4713F200567","9B3702B0E788C6D FE3C9786A","05E566DF","J JPG",32768 " AFA836117B1B572FAE4713F200567","9B3702B0E788C6D FE3C9786A","05E566DF","J JPG",32768 " AFA836117B1B572FAE4713F200567","9B3702B0E788C6D FE3C9786A","05E566DF","J JPG",32768 "SHA-1","MD5","CRC32","FileName","FileSize","ProductCode",\\ "OpSystemCode","SpecialCode" " EDD92C4E3D2E F849",\\ "392126E756571EBF112CB1C1CDEDF926",\\ "EBD105A0","I05002T2.PFB",98865,3095,"WIN","" Harald Baier Hash Functions in Forensics / WS 2011/ /41 Whitelisting: Anti-detection 1. Task: Find a suspicious file s with h(s) = h(g): Attacker has to solve the second preimage problem for some g G. As of today hardly possible (even for MD5). If RDS is used, both SHA-1 and MD5 preimages have to be found. 2. Attack the whitelist: Breach integrity and authenticity of whitelist. Example: Attack the distribution channel. If RDS is downloaded via http, spoofing is possible.

12 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Whitelisting: Assessment 1. General assessment: Well-known and established process in computer forensics. If database is trusted, no false positives (positive = benign). 2. Possible bottleneck: Size of database. Size of database is increasing. Currently RDS is about 6 gigabyte. Harald Baier Hash Functions in Forensics / WS 2011/ /41 Blacklisting 1. Underlying idea: Generate a database of known to be bad files. Let B denote this set. Find automatically a suspicious file on base of its fingerprint, which matches a fingerprint of a file in B. 2. Sample suspect files: Malware. Encryption or steganographic software. Corporate secrets. IPR protected files. Child pornography.

13 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Blacklisting: Evaluation 1. Anti-detection approach: Let a suspicious file b B be given. Change some (irrelevant) bit of b to get b. Consequence: h(b ) is very different from h(b). b is not detected automatically. 2. Core problem: Cryptographic requirements of a hash function and forensic goals are complementary. A suspicious file similar to an element of B is not detected. 3. Fragments of elements of B are not identified, too. Harald Baier Hash Functions in Forensics / WS 2011/ /41 Fuzzy Hashing

14 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Goals 1. Overcome drawbacks of cryptographic hash functions in the context of computer forensics. 2. Main drawbacks are: Data acquisition: Integrity of copy is destroyed, if some bits change. White-/Blacklisting: Suspect files similar to known to be bad files are not detected. Fragments are not detected (due to deletion, fragmentation). 3. Currently known approaches: Segment hashes (also called block hashes): Tool dcfldd. Context-triggered piecewise hashes: Tool ssdeep. Similarity digests: Tool sdhash. Harald Baier Hash Functions in Forensics / WS 2011/ /41 Segment Hashes 1. Underlying idea: Split input data (volume, file) in blocks of fixed length. Compute for each segment its cryptographic hash. Lookup in hash database for matches. 2. Original aim: Improve integrity of storage media.

15 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Segment Hashes: Example from NIST 1. Sample tool of Nicholas Harbour (since 2002): dcfldd: An extension of dd. Department of Defense Computer Forensics Laboratory. Provides MD5, SHA-1, SHA-2 family. 2. Evaluation by NIST (Douglas White, 2008): Hashing of File Blocks: When Exact Matches are not Useful. NIST worked on Windows 2000 and XP OS files. Main result: File-based data reduction leaves an average 30% of disk space for human investigation. Incorporating block hashes reduces this to an average of 15%. Assist in recognising wiped media. Harald Baier Hash Functions in Forensics / WS 2011/ /41 Example: Save first partition of a HDD Chain of custody preserved $ dcfldd if=/dev/hda1 of=image-hda1.dd bs=512 hashwindow=4096 hash=sha : da0bd2b16c7cd5acb5695e9d81fb6d832cba85312d87e08d0c675e41b608de : 281f4b8ac2dcda0f3fd9a0642a694fedf829d7567a531b1cfc8925f94eebe7a : 1c05a3c7251b666c1ec4a2b689e25f95a92a311613ce685fdf7cbf e : 6c9c17f271f18587bcccf8f9c6154b4bff764664a3eb8ddf c5c4698b : c61cd658e73450dfb0dfc9a1d83cdbddd162d9194d81f27f0516bb107280e841 [REMOVED] $ dcfldd if=image-hda1.dd of=/dev/null bs=512 hashwindow=4096 hash=sha : da0bd2b16c7cd5acb5695e9d81fb6d832cba85312d87e08d0c675e41b608de : 281f4b8ac2dcda0f3fd9a0642a694fedf829d7567a531b1cfc8925f94eebe7a : 1c05a3c7251b666c1ec4a2b689e25f95a92a311613ce685fdf7cbf e : 6c9c17f271f18587bcccf8f9c6154b4bff764664a3eb8ddf c5c4698b : c61cd658e73450dfb0dfc9a1d83cdbddd162d9194d81f27f0516bb107280e841 [REMOVED]

16 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Example: Save first partition of a HDD Chain of custody partly destroyed $ dcfldd if=/dev/hda1 of=image-hda1.dd bs=512 hashwindow=4096 hash=sha : da0bd2b16c7cd5acb5695e9d81fb6d832cba85312d87e08d0c675e41b608de : 281f4b8ac2dcda0f3fd9a0642a694fedf829d7567a531b1cfc8925f94eebe7a : 1c05a3c7251b666c1ec4a2b689e25f95a92a311613ce685fdf7cbf e : 6c9c17f271f18587bcccf8f9c6154b4bff764664a3eb8ddf c5c4698b : c61cd658e73450dfb0dfc9a1d83cdbddd162d9194d81f27f0516bb107280e841 [REMOVED] $ dcfldd if=image-hda1.dd of=/dev/null bs=512 hashwindow=4096 hash=sha : da0bd2b16c7cd5acb5695e9d81fb6d832cba85312d87e08d0c675e41b608de : 281f4b8ac2dcda0f3fd9a0642a694fedf829d7567a531b1cfc8925f94eebe7a : 1c05a3c7251b666c1ec4a2b689e25f95a92a311613ce685fdf7cbf e : f361bc439d0fc4215eba5523cba663c5371d47c358d68b a29e972e : c61cd658e73450dfb0dfc9a1d83cdbddd162d9194d81f27f0516bb107280e841 [REMOVED] Harald Baier Hash Functions in Forensics / WS 2011/ /41 Segment Hashes: Evaluation 1. Anti-Blacklisting is very easy: Introduce an irrelevant byte in the first sector. All segment hashes differ from the stored segment hashes. Modified suspect file is not detected. Demo. 2. A good technique for whitelisting (see NIST results). 3. Size of segment hash database is large: 4096 byte block size, SHA-1. size of hash database size of raw data = = = 1 terabyte of raw data yields a 5 gigabyte hash database. 4. Hash database depends on the hashwindow size.

17 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Context Triggered Piecewise Hashes 1. Underlying idea: Split input data (volume, file) in blocks of variable length. The end points of the blocks are determined by a rolling hash: Its value only depends on the current context. A window (e.g. of size 7 bytes) slides over the input. Context = Bytes of input data in the current window. If rolling hash matches a trigger value, an end point is set. Rolling hash function is assumed to be pseudo random: P F. Compute for every segment its cryptographic hash. The sequence of these segment hashes is the context triggered piecewise hash (CTPH) of the input. Lookup in hash database for matches. Harald Baier Hash Functions in Forensics / WS 2011/ /41 Context Triggered Piecewise Hashes 1. Originally proposed for spam detection (spamsum by Andrew Tridgell, 2002) 2. Ported to forensics by Jesse Kornblum, 2006: ssdeep.

18 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Piecewise Hashes: The algorithm Harald Baier Hash Functions in Forensics / WS 2011/ /41 CTPH: A sample tool 1. ssdeep (based on spamsum). 2. Window size is CTPH is a sequence of printable characters: Only the least significant 6 bits (LS6B) of a segment hash are considered. LS6B are encoded base ssdeep decides about a match: On base of the edit distance (changing, inserting,... characters) of two CTPHs. 5. Demo. Edit distance is rescaled to a percentage match score.

19 Harald Baier Hash Functions in Forensics / WS 2011/ /41 CTPH: Sample Research Questions 1. Rolling hash: Shall be efficient and pseudo random. Current implementation is fast, but not pseudo random. Task: Find a slightly slower, but pseudo random rolling hash. 2. Fragment detection: Kornblum s approach fails, if fragments are much smaller than the original file. Task: Find a different approach. 3. Edit distance: Kornblum s approach addresses text files (due to spam detection). Task: Find a more general approach, which also addresses images, videos,... Harald Baier Hash Functions in Forensics / WS 2011/ /41 Fuzzy Hashing Fuzzy Hashing

20 Harald Baier Hash Functions in Forensics / WS 2011/ /41 Vision Fuzzy Hashing 1. Find a similarity preserving hash function. Fuzzy hash function, denoted by f. m and m are similar = f(m) and f(m ) are similar, too. 2. Find a general (forensic) metric to measure similarity: Let d : {0, 1} {0, 1} R + 0 be such a metric. d shall be applicable to txt, doc, odt, jpg, bmp, devices, f is a function with d(m 1, m 2 ) d(f(m 1 ), f(m 2 )). Harald Baier Hash Functions in Forensics / WS 2011/ /41 Fuzzy Hashing Fuzzy Hashing: Applications 1. Forensics (on the file level): Detect similar files. Blacklisting: Detect manipulated suspicious files. Find fragments of suspicious data. Whitelisting: Find changed unsuspicious files. 2. Biometrics: Template protection. 3. Malware: Detect obfuscated malware (e.g. metamorphic malware). 4. Junk mail detection.

21 Fuzzy Hashing Questions? Harald Baier Hash Functions in Forensics / WS 2011/ /41