Digital Meta-Forensics: Quantifying the Investigation. Richard E Overill and Jantje A M Silomon

Transcription

1 Digital Meta-Forensics: Quantifying the Investigation Richard E Overill and Jantje A M Silomon Department of Computer Science, King's College London, Strand, London WC2R 2LS, UK {richard.overill jantje.a.silomon}(at)kcl.ac.uk Abstract We review, analyse and evaluate recent developments in two related areas of digital forensics. The first involves quantifying the extent to which the recovered digital evidential traces support the prosecution s contention that a particular digital crime has been committed. The second addresses the issue of quantifying the cost-effectiveness of the digital forensic investigative process, in order to optimise the deployment of valuable and scarce resources for maximum efficacy. Keywords: conditional probability; Bayesian network; likelihood ratio; odds ratio; complexity; cost-effectiveness; return on investment; cost-benefit ratio; forensic triage. 1. Introduction and Background Digitial forensic analysis has up to now not kept pace with conventional forensic analysis in the matter of quantifying the degree of certainty with which a hypothesis and a corresponding set of evidential traces can be causally linked. For example, in the case of fingerprint or DNA analyses the probability of two distinct individuals exhibiting a match is known and hence the plausibility of the hypothesis can be determined quantitatively in the form of a likelihood ratio. In the context of digital forensic evidence however, qualitative statements, such as "very likely" are routinely used by forensic examiners and expert witnesses. Not surprisingly, defence lawyers assigned to digital crime cases have become aware of this discrepancy and have attempted to exploit it to persuade the court that the prosecution does not possess evidence of sufficient probative value. However, the development of potentially suitable methodologies and techniques for the quantitative interpretation of digital forensic investigations is underway [1-4] and offers the prospect of bringing a degree of numerical certitude to the recovered evidence in such cases. At the same time, metrics for analysing the cost-effectiveness of digital forensic investigations are also being developed [5, 6], as are their implementation in the form of practical software solutions that can be used by digital forensic technicians to perform a type of forensic triage [7]. These developments are particularly significant in view of the current severe shortage of experienced digital forensic examiners and the accelerating increase in digital crimes, all in the context of a global economic recession. In this paper we review the latest developments in both of the areas referred to above and additionally highlight those issues that remain problematic or have yet to be fully addressed.

2 2. Concepts and Methodologies 2.1 Metrics for Plausibility The remainder of this subsection describes a top-down and a bottom-up approach to quantifying the degree of plausibility of a (prosecution or defence) hypothesis regarding the putative sequence of events in relation to the recovered digital evidential traces. A top-down approach makes use of Bayesian conditional probabilities, i.e. the probability of recovering the evidential traces E given that the hypothesis H is correct: Pr(E H). A bottom-up approach, on the other hand, makes use of Bayesian posterior probabilities, i.e. the probability that the hypothesis H is correct given that the evidential traces E have been recovered: Pr(H E). The Likelihood Ratio (LR) is a top-down plausibility metric, which is used routinely by conventional forensic scientists (see, for example, [8, 9]). The generalised LR [10] is given by: Λ = Pr(E H) / Pr(E H c ) where H c is the logical complement of H. In the context of a Bayesian network representation of an actual digital crime investigation (see, for example, [1, 2] and Figure 1), the numerator of Λ is obtained by setting the value of the main hypothesis to true and then multiplying together the resulting probabilities for each of the evidential traces, under the assumption that the evidential traces are mutually independent; similarly, setting the value of the main hypothesis to false yields the denominator. The Odds Ratio (OR), on the other hand, is a bottom-up plausibility metric, which is also widely used by forensic scientists [9]. The OR is given by: Ο = Pr(H E) / Pr(H c E) and cannot be obtained directly from a Bayesian network representation. Instead, a complexity based model such as the Operational Complexity Model (OCM) [3] can be used to estimate the work (both computational and human) involved in producing the observed evidential traces E via the route specified by H, and also via all the remaining enumerated feasible routes satisfying H c. In the OCM, computational complexity [11] and GOMS-KLM [12] represent the computational and human contributions respectively. A major conceptual step of the OCM is the postulate that the probability of a given feasible route k (as specified by hypothesis H k ) is inversely proportional to its complexity, C k : Pr(H k E) C k -1 where the proportionality constant α is uniquely determined by the normalisation condition on the sum of the probabilities over all feasible routes: α = ( C k -1 ) -1 Thus, by enumerating all feasible routes k leading to the formation of the observed evidential traces E and evaluating their respective complexities C k, a numerical estimation of the OR for any one particular route over all other feasible routes can be obtained.

3 It should be noted that although a simple proportionality relationship exists between LR and OR: Ο Λ, the constant of proportionality in this case is: β = Pr(H) / Pr(H c ) that is, the (prior) odds of H, which is not usually known. For this reason LR and OR are not normally inter-convertible. 2.2 Metrics for Cost-effectiveness With law enforcement resources (principally manpower, money and time) already overstretched in relation to the number of digital forensic investigations requested, it becomes important to develop methods of determining whether or not any particular investigation is worthwhile undertaking. This leads naturally to the concepts of forensic triage and prioritisation. A preliminary filtering or prescreening phase will enable evidentially hopeless investigations to be abandoned quickly, and the remainder to be ranked in probable order of evidential strength. A number of criteria have been proposed for making the initial assessment, including cost-efficiency [5] and return-on-investment (ROI) [6]. Cost-efficiency ranks the recovered evidence against the expected evidence for a known type of crime. A forensic technician is guided through the assessment by a software application (for example, Digital Forensic Advisor [7]) in a pre-determined sequence which seeks out the most evidentially significant traces first, and, among traces of equal evidential weight, the lowest cost traces first [5]. In this way, an evidentially hopeless assessment can be detected early on and abandoned, while only those assessments which exceed a pre-determined evidential threshold (for example, 80% of the expected evidence) are passed on to an experienced forensic examiner for full processing. Return-on-Investment is a familiar business decision making concept which is inversely related to the equally familiar cost-benefit ratio (CBR). If the evidential weight of a given trace i is w i and its corresponding cost is c i then its ROI is given by w i / c i and its CBR by c i / w i. The evidential traces for a known type of crime can be ranked in order of descending ROI (or of ascending CBR) and then assessed in that order. Refinement of this scheme is possible: for example, where a particular trace i contributes to n i >1 evidentiary chains then its effective weight is given by w i = n i w i and the effective ROI is then (n i w i ) / c i [6]. Alternative ranking metrics, such as n i + (c i /10) and (n i /c i ), have also been proposed [6]. Each metric yields a subtly different assessment ranking for the evidentiary traces (see [6], Table 2). A digital forensic laboratory s choice of which ranking to adopt will depend on its individual priorities (for example, throughput versus resources) as well as on the nature of the suspected digital crime (for example, civil versus criminal, or large scale versus small scale).

4 3. Results and Discussion The conceptual frameworks outlined in Section 2.1 above have been applied to the evidential traces from two actual prosecuted digital crimes. 3.1 Top-down analysis In [2], the LR for an online auction fraud case was calculated under the assumption that there are just two feasible routes by which the recovered digital evidential traces could have been produced, corresponding to the prosecution s (criminal) hypothesis H p and the defence s (non-criminal) hypothesis H d. Note that, since the recovered digital evidential traces are not in dispute per se, the Bayesian networks for the two hypotheses differ only in the interpretation of the significance of the traces as reflected in their respective sub-hypotheses. In this case, the calculation yields a LR for H p over H d of 164,000, which can be qualitatively interpreted as indicating very strong evidentiary support for H p over H d [8]. This finding may be taken by the prosecuting authority as one indication inter alia that it is worthwhile proceeding to trial. It should however be pointed out here that this type of calculation may also be employed by the defence side in an adversarial judicial system in an attempt to improve or tune their own hypothesis so that it compares more favourably with the prosecution s. Another approach that the defence might wish to adopt is to challenge the validity of the individual conditional probability values Pr(E i H j ) embedded in the prosecution s Bayesian network. Each value represents the probability that the evidential trace E i is found given that the sub-hypothesis H j is true. These values may be obtained by aggregating the responses of a group of experienced digital forensic examiners to a carefully constructed questionnaire [1]. In order to study the dependence of a Bayesian network on its embedded evidential conditional probabilities, a rigorous sensitivity analysis [4] has been performed on the Bayesian network used for investigating illegal peer-to-peer (P2P) file-sharing over a BitTorrent (BT) network [1]. While simultaneously replacing every aggregated conditional probability with its minimum response value was found to produce the largest change in the Bayesian network ( 25.5%), the chances randomly selecting minimum values for all 18 evidential conditional probabilities is negligible (of order ). However, systematically replacing any one conditional probability consistently produced changes of less than 2.6% in magnitude [4]. The implication of this finding is that the BT network, and, by extension, other similar Bayesian networks for investigating digital crimes, is not particularly sensitive to the precise values employed for each of the evidential conditional probabilities. A similar observation holds with regard to missing (that is, unrecovered) evidential traces. More often than not the complete set of expected digital traces from a suspected crime is not recovered, and the defence might wish to challenge the strength of the prosecution case on these grounds. The BT sensitivity analysis [4] also studied the effect on the Bayesian network of systematically removing each individual trace, all possible pairs of traces, and a selection of groups of three or more traces. The maximum change due to one or two missing traces (out of a possible total of 18 traces for BT) was 10.3%, and the sampled cases involving three and four missing traces also fell within this bound. Taken together, these findings effectively block potential challenges on the grounds of conditional probability values or missing evidential traces.

5 3.2 Bottom-up analysis In [3], the OR for the illegal P2P file-sharing (BT) investigation was calculated under the assumption that the Trojan defence [13] was the only other feasible route to the recovered evidential traces. This approach employed the non-bayesian OCM [3], requiring the enumeration of the human and computational complexities of each step of both feasible routes. A further assumption was made that the user of the seized computer frequently participated in online filesharing. The size of the illegally shared multimedia file was taken as 4GB. In this case the initial calculation yielded an OR of 4.60 in favour of the prosecution hypothesis. However, the initial model assumed that no up-to-date anti-malware defence was operational on the seized PC. Inclusion of an anti-malware defence with a typical Trojan detection probability of 0.98 raised the OR to 277, or, put another way, the probability of the prosecution hypothesis is if the Trojan defence is the sole alternative hypothesis. This is by no means the final word on the topic of calculating the OR ab initio. It is possible to weight the human and computational complexity terms to reflect the difference in cycle times between a human brain and a PC (of order 10 7 ). Disk access, which is of order 10 5 slower than RAM access, may also be included in the model in a straightforward manner [3]. 4. Summary and Conclusions In the first instance, the quantitative analysis methods described in this paper would most naturally be employed in-house by forensic investigators to assess the likely probative value of the recovered evidence, and to decide whether or not a full digital forensic investigation is warranted. At a later stage, they could be adopted by the relevant prosecution authority (e.g. the Crown Prosecution Service of England and Wales) in reaching a decision as to whether or not to mount a prosecution given the evidence recovered from the full digital forensic investigation. Finally, they might be used within trial proceedings (possibly by both prosecution and defence counsels in an adversarial judicial system) as an additional means of persuading the court of the respective strengths and weaknesses of their own and their opponent s cases. For perfectly good reasons judicial systems tend to be inherently conservative and it is to be expected that there would be a natural time lag (of perhaps at least a decade) between each of these three phases of development.

6 References [1] Kwan M, Chow K-P, Law F & Lai P, Reasoning About Digital Evidence Using Bayesian Networks, Advances in Digital Forensics IV, Ch.12, pp , Springer (2008). [2] Kwan Y K, Overill R E, Chow K P, Silomon J A M, Tse H, Law Y W & Lai K Y, Evaluation of Evidence in Internet Auction Fraud Investigations, Proc.6th Annual IFIP WG 11.9 International Conference on Digital Forensics, Hong Kong, 3-6 January 2010, Advances in Digital Forensics VI, Ch.7, pp , Springer (2010). [3] Overill R E, Silomon J A M & Chow K-P, A Complexity Based Model for Quantifying Forensic Evidential Probabilities, Proc.3rd International Workshop on Digital Forensics (WSDF 2010), Krakow, Poland, February 2010, pp [4] Overill R E, Silomon J A M, Kwan Y K, Chow K-P, Law Y W & Lai K Y, Sensitivity Analysis of a Bayesian Network for Reasoning about Digital Forensic Evidence, Proc.4th International Workshop on Forensics for Future Generation Communication environments (F2GC-10), Cebu, Philippines, August 2010 (to appear). [5] Overill R E, Kwan Y K, Chow K-P, Lai K Y & Law Y W, A Cost-Effective Digital Forensics Investigation Model, Proc. 5th Annual IFIP WG 11.9 International Conference on Digital Forensics, Orlando, Florida, USA, January 2009, Advances in Digital Forensics V, Ch.15, pp , Springer (2009). [6] Cohen F, Two Models of Digital Forensic Analysis, Proc.4th International IEEE Workshop on Systematic Approaches to Digital Forensic Engineering (SADFE-2009), Oakland, CA, 21 May 2009, pp [7] Digital Forensic Advisor, a software application developed jointly by King's College London and Hong Kong University, funded by Innovation China UK (June 2010). [8] Keppens J, Towards Qualitative Approaches to Bayesian Evidential Reasoning, Proc.11th ACM International Conference on Artificial Intelligence and Law (ICAIL 07), Stanford, CA, 4-8 June 2007, pp.17 25, Table2. [9] Lucy D, Introduction to Statistics for Forensic Scientists, Wiley, Chichester, UK (2005). [10] Evett I, Establishing the evidential value of a small quantity of material found at a crime scene, J For Sci Soc, 33(2) (1993) [11] Papadimitriou C H, Computational Complexity, Addison-Wesley, Reading, MA (1994). [12] Kieras D, Using the Keystroke-Level Model to Estimate Execution Times, University of Michigan (2001), available online at: [13] Haagman D and Ghavalas B, Trojan Defence: A Forensic View, Digital Investigation, 2 (2005)

7 HYPOTHESES: H The seized computer was used as the initial seeder to share the pirated file on a BitTorrent network H 1 The pirated file was copied from the seized optical disk to the seized computer H 2 A torrent file was created from the copied file H 3 The torrent file was sent to newsgroups for publishing H 4 The torrent file was activated, which caused the seized computer to connect to the tracker server H 5 The connection between the seized computer and the tracker server was maintained EVIDENCE: E 1 Modification time of the destination file equals that of the source file E 2 Creation time of the destination file is after its own modification time E 3 Hash value of the destination file matches that of the source file E 4 BitTorrent client software is installed on the seized computer E 5 File link for the shared file is created E 6 Shared file exists on the hard disk E 7 Torrent file creation record is found E 8 Torrent file exists on the hard disk E 9 Peer connection information is found E 10 Tracker server login record is found E 11 Torrent file activation time is corroborated by its MAC time and link file E 12 Internet history record about the publishing website is found E 13 Internet connection is available E 14 Cookie of the publishing website is found E 15 URL of the publishing website is stored in the web browser E 16 Web browser software is available E 17 Internet cache record about the publishing of the torrent file is found E 18 Internet history record about the tracker server connection is found Figure 1 BitTorrent Network Diagram [1]