IBM Research Report. Towards the Open Advancement of Question Answering Systems

Size: px
Start display at page:

Download "IBM Research Report. Towards the Open Advancement of Question Answering Systems"

Transcription

1 RC24789 (W ) April 22, 2009 Computer Science IBM Research Report Towards the Open Advancement of Question Answering Systems David Ferrucci 1, Eric Nyberg 2, James Allan 3, Ken Barker 4, Eric Brown 1, Jennifer Chu-Carroll 1, Arthur Ciccolo 1, Pablo Duboue 1, James Fan 1, David Gondek 1, Eduard Hovy 5, Boris Katz 6, Adam Lally 1, Michael McCord 1, Paul Morarescu 1, Bill Murdock 1, Bruce Porter 4, John Prager 1, Tomek Strzalkowski 7, Chris Welty 1, Wlodek Zadrozny 1 1 IBM Research Division Thomas J. Watson Research Center P.O. Box 704 Yorktown Heights, NY Carnegie Mellon University 3 University of Massachusetts 4 University of Texas 5 USC/ISI 6 MIT 7 SUNY Albany Research Division Almaden - Austin - Beijing - Cambridge - Haifa - India - T. J. Watson - Tokyo - Zurich

2 Towards the Open Advancement of Question Answering Systems December 2008 David Ferrucci (IBM), Eric Nyberg (Carnegie Mellon University), James Allan (University of Massachusetts), Ken Barker (University of Texas), Eric Brown (IBM), Jennifer Chu-Carroll (IBM), Arthur Ciccolo (IBM), Pablo Duboue (IBM), James Fan (IBM), David Gondek (IBM), Eduard Hovy (USC/ISI), Boris Katz (MIT), Adam Lally (IBM), Michael McCord (IBM), Paul Morarescu (IBM), Bill Murdock (IBM), Bruce Porter (University of Texas), John Prager (IBM), Tomek Strzalkowski (SUNY Albany), Chris Welty (IBM), Wlodek Zadrozny (IBM) 1 Summary On February 27-28, 2008, a group of researchers from industry and academia met to discuss the state of the Question Answering (QA) field. The discussion focused on recent experiences from funded research programs (e.g. AQUAINT, HALO) and open evaluations (e.g. TREC, NTCIR). The group acknowledged that funded research programs and evaluations have been instrumental in establishing fundamental QA research. However, major advances in the field of QA are yet to be realized. Advances that can openly accelerate progress and greatly generalize QA technologies to produce scalable, more adaptable methodologies and business applications are within our reach. Although there are deep technical challenges, we believe these challenges can be effectively addressed by open collaboration in the open-source development of integrated QA technologies aimed at well-chosen sets of QA applications. It is currently difficult to discern the adaptability or generality of algorithms that are published as part of a solution to a particular QA problem. There are no means to leverage individual contributions from distributed groups, so development tends to take place in a single organization, with lots of redundant (as opposed to shared) effort. As a result, sponsors have difficulty in determining which component technologies are really working, which require more research attention, and which are already working well enough for the problem at hand. Attempts to adapt systems that were built and evaluated using TREC datasets for use in other applications indicate that there is much fundamental research left to be done, particularly in improving the run-time speed, answer confidence, and domain adaptability of QA technologies. The goal of the workshop was not to write another general, all-encompassing road map for QA research problems, but to address the problems mentioned above, so that ongoing research can be streamlined reducing overall cost and time to innovate, while providing a more supportive environment for individual research contributions from academic organizations. The concept that emerged from the workshop is one of open advancement: the use of shared system models, open-source components, collections of challenge problems and common evaluation metrics, so that the contribution of each technology to end to end performance can be accurately measured and the community as a whole can uniformly advance system performance on an ever broadening range of QA problems. 1

3 Our specific objective is to combine formal metrics and rigorous module and end-to-end system evaluation with a collaborative research process that allows our field, as a research community, to achieve monotonically increasing performance levels across multiple instances of QA problems, while managing overall research and development cost effectively. This document captures the thoughts and proposals from the workshop regarding open advancement, and is intended as a working document to aid in the creation of new sponsored research programs in industry and government. 2 Introduction Question Answering (QA) is an application area of Computer Science which attempts to build software systems that can provide accurate, useful answers to questions posed by human users in natural language (e.g., English) 1. A QA system is a software system that provides exact answers to natural language questions for some range of topics. The notion of exact in this context is ultimately a subjective measure intended to indicate that a QA system is distinguished by providing responses that contain just the information necessary to precisely answer the question intended by user. The QA system s exact answer may be supplemented with additional information, including a justification or dialog explaining why the provided answer is correct. Another distinguishing and important characteristic of QA systems is that their accuracy and their ability to justify an answer should increase with the amount of relevant information provided in the question. They should, therefore respond more accurately and more completely to longer, denser questions, that is, to questions which provide more information about what is being asked. This behavior would suggest that QA systems rely on a deeper semantic understanding of the intent of the question. The resources used to answer questions can vary from unstructured data (e.g., typical web pages, blog posts) and semi-structured data (e.g., Wikipedia) to completely structured data (facts mined from the Web or pre-existing databases). Research and development of QA systems has been evaluated since 1999 in the yearly TREC QA track evaluations conducted by NIST, and has been supported in the U.S. by the AQUAINT program ( ). Question answering systems are also evaluated in the context of two other academic workshops, CLEF (Europe) and NTCIR (Asia). The AQUAINT program has already demonstrated the practical potential of question answering software. Several teams in the program have delivered working end-to-end systems to the program sponsor, and in some cases systems have been deployed and 1 In this paper we limit our discussion to text-based QA, but acknowledge that QA systems can be developed to directly address other modalities including image, speech, music and video for example. 2

4 made to interoperate via web services. Mature systems tuned to TREC QA factoid questions have delivered batch answers with 70% first-answer accuracy. Tempering these successes, recent TREC experience has also shown that system performance tends to drop dramatically when a new domain or problem (question type) is introduced, which might imply that systems are over-tuned to a particular problem instance for each yearly evaluation, and that research progress has been incremental rather than general. If one considers the TREC scores for a particular system across yearly tasks, one notes that even with constant research and development, teams cannot always sustain the same level of performance on a new task. The superficial level of algorithmic detail presented in a typical academic paper on a complex QA system does not support direct checking and replication of results, so it is difficult to assess the contribution of individually reported algorithms to overall system performance. In a 2003 document entitled Issues, Tasks and Program Structures to Roadmap Research in Question & Answering (Q&A), Burger et al. presented a diverse set of research topics in QA along with a corresponding set of planned evaluations (e.g. TREC QA tracks or tasks), but without details on how metrics would be developed and applied for more detailed forms of QA evaluation. In a workshop on QA evaluation held at LREC in 2002 (Maybury, 2002), the participants spent time creating considerable road map detail regarding resources for development and evaluation, methods and algorithms, and systems performance, but neglected to outline the process of collaborative systems integration, measurement and analysis. Although prior workshops and road mapping exercises in QA have helped to define the agenda for advanced research, less progress has been made on the definition of a transparent, repeatable process that can be used to evaluate component technologies and end-to-end systems working on a variety of problems. Our observation is that precisely such a transparent, repeatable process, and a variety of shared evaluation metrics is what is required to help QA technology reach more general levels of applicability and higher levels of sustained and relevant performance. The process should be general enough to support not only question answering from English text resources, but also video and audio retrieval, and cross-lingual retrieval from resources in other languages. At the conclusion of the workshop, the participants agreed to collaborate on a white paper to document the vision and requirements for open advancement of QA systems. This document is the result of that collaboration, and contains these main sections: Vision. What is open advancement? Why is it important? What are the benefits for each stakeholder (industry, academia, government, etc.)? Challenge Problems. We outline a set of challenge problems (QA applications) that will drive improvements in the state of the art across measurable dimensions (such as answer accuracy, answer confidence, and response time); the problems are diverse by design, so that performance gains can be tested across multiple problem instances to demonstrate general advancement. 3

5 Approach to Open Collaboration. A discussion of how a consortium of researchers from industry, academia and government could work together towards open advancement of QA technology. Open Collaboration Model. An outline of the intended software development process, a first version of which has already been established by IBM and is being tested at Carnegie Mellon University. 3 Vision A typical question answering system is a complex software configuration which combines a variety of text processing modules into a particular dataflow configuration. Two broad objectives of QA research are: a) to identify the most useful text processing modules for acquiring knowledge, analyzing questions, documents, passages, answers, etc.; and b) to identify the most effective dataflow configuration for a run-time QA system. A particular QA technology is comprised of a set of modules and dataflows representing a specific approach or solution, usually tailored to a particular type of question or resource. When the application domain requires a very fast system response to an input question, an additional objective is to design and deploy an effective run-time hardware configuration which takes advantage of parallel distributed computing. The goal of open advancement is to rapidly accelerate the pace of progress, both in developing effective core algorithms and in deploying practical solutions. We plan to establish a broader collaboration on the open advancement of QA that will help to share innovation across the field as a whole, and provide deeper insight into the generality and domain adaptability of emerging QA technologies. The open advancement of QA is expected to promote teaming of stakeholders from industry, academia and government. Overall, we expect open advancement to provide the following synergies between stakeholder groups: Industrial partners gain better insight into levels of component and system performance required to solve practical problems, and are better able to leverage commercial applications of QA technology; Government partners gain better insight into performance bottlenecks and measures that can be used to associate new research efforts with detailed performance evaluation results; as a result, research program management becomes more cost-effective; Academic partners gain the opportunity to evaluate new component algorithms in the context of end-to-end systems working on a variety of problem instances, improving the breadth and quality of their research output without having to build a complete end-to-end framework on their own. The broader collaboration will be supported by a group consensus on best practices regarding key elements of QA system development: a) a shared logical architecture for 4

6 integration and evaluation; b) a shared deployment architecture for run-time systems; c) a set of common challenge problems to drive innovation along important dimensions; d) a collaborative, distributed process for integration, testing and evaluation; and e) an open collaboration approach to software development, to support transparent, controlled evaluation and straightforward replication of results. These aspects of open advancement are introduced in the subsections that follow. 3.1 Shared Logical Architecture If one goal of QA research is to identify the most effective algorithms and dataflows for a given problem, then the notion of effective must be defined via some set of formal metrics on QA system modules and dataflows. Metrics are applied to a particular dataflow, corpus and question set in order to precisely measure module and system performance. As the system is tested across a broad variety of applications, more data are gathered, and over time greater insight is gained into how to select the best components and configure optimal dataflows for practical applications. If these fundamental tenets of system engineering can be applied to QA software development, it will become possible to relate investment in particular text processing technology with actual performance gains or losses in practical application. To support this vision of shared modules, dataflows and evaluation measures, an open collaboration will include a shared logical architecture a formal API definition for the processing modules in the QA system, and the data objects passed between them. For any given configuration of components, standardized metrics can be applied to the outputs of each module and the end-to-end system to automatically capture system performance at the micro and macro level for each test or evaluation. 3.2 Shared Deployment Infrastructure The logical architecture will be complemented by a shared run-time evaluation framework, which allows particular configurations of components to be formally evaluated on particular problem instances. Another important advantage of a shared logical architecture is that it enables the development of parallel, distributed computing infrastructure for large-scale deployment of system components. By sharing the same logical architecture, all teams will be able to take advantage of distributed computing frameworks and resources provided by other partners. IBM and CMU have begun to collaborate on building an open-source framework for OAQA. Elements of the framework that are contemplated or already in development include: A Java implementation of a logical architecture for OAQA (data structures, components, configurations, resources, etc.); Tools to configure QA components into a particular end-to-end configuration; Tools to manage the execution of end-to-end configurations on distributed commodity hardware; Tools to capture execution results, automatically evaluate them against known answer keys, and compare current performance to past performance at the module and system level. 5

7 IBM is also considering providing access to specialized high-performance computing resources for the open advancement of QA. Of particular interest are map-reduce-style algorithms for retrieval, answer extraction and answer validation on massively parallel systems like BlueGene. 3.3 Challenge Problems Each QA application domain has certain characteristics related to the difficulty of the language(s), resource structure, questions, expected answers, etc. More general algorithmic approaches are less sensitive to variations in these characteristics, and can be adapted to new problem instances more readily; more language- and domain-specific approaches are much more sensitive to variations in language and resource structure, and cannot be easily adapted to completely new problem instances. An effective challenge problem is one which cannot be solved without a significant improvement in QA system performance along some dimension of measurement (such as question difficulty, answer confidence, answer accuracy, system response time, etc.). By formulating a series of challenge problems across different information domains, it will become possible to drive core research and development and enhance domain adaptability in the longer term. Section 4 contains a detailed discussion of the different dimensions of a challenge problem, with examples drawn from particular QA problem domains. 3.4 Collaborative Process During the workshop, we identified three main areas for collaboration across industry, government and academia: Challenge Problems & Metrics. Challenge problems that drive the innovation to provide better business solutions, and the metrics that measure progress against those challenges, should be defined by representatives from all three groups. Core Technologies. Identifying and standardizing the use of different core technologies (for example, document retrieval, NLP tools) will help to advance the entire field of research, as well as drive forward our ability to deliver higher quality, lower-cost software solutions. Infrastructure and Evaluation. By designing and building a shared infrastructure for system integration and evaluation, we can reduce the cost of interoperation and accelerate the pace of innovation. A shared logical architecture also reduces the overall cost to deploy distributed parallel computing models to reduce research cycle time and improve run-time response. All three areas will involve representatives from industry, academia and government, but it is expected that academic partners will wish to focus on technology research and evaluation, industrial partners will support frameworks for integration and evaluation, 6

8 and external sponsors will provide guidance regarding appropriate challenge problems and how they map to application systems that will meet current operational goals. The open collaboration will include a variety of shared activities; each activity will be associated with a shared process that is a agreed upon by the collaborators. The shared activities will include: a) defining a shared logical architecture; b) defining a set of common metrics for modules and end to end systems; c) establishing a framework to support integration and evaluation of end-to-end systems and modules; d) defining a set of challenge problems to be undertaken; and e) defining a shared process for ongoing integration, testing and refinement. These activities are discussed in more detail in Section Open Collaboration Model Open advancement is based on transparency of evaluation and reproducibility of results. The collaborative process will best be served by an open-source development model, where newly-created code is placed into open source for direct examination by all partners. This will help to ensure that the work is understandable, repeatable and reusable. Examples of successful open-source projects that have helped accelerate the development and application of key technologies include Lemur (document indexing and retrieval with language models), Nutch (web search engine), Hadoop (distributed file system with support for map-reduce), Lucene (general-purpose Java search engine) and UIMA (a framework for content analysis and automatic annotation), just to name a few. Pre-negotiation of open-source teaming arrangements also helps to expedite research contracting with academic institutions. The role of open-source development is discussed in more detail in Section 6. A compelling example of the success of this model is the annual RoboCup competition, which evaluates the soccer-playing skills of robot players in an open international forum 2. RoboCup has steadily reduced the gap between robot and human performance over the past 10 years, in part because all participants are required to publish their code after each competition [Asada, et al, 2007]. Conversely, a closed model, in which competitive evaluations are conducted without a requirement for openness, reproducibility, and transparency, has hampered scientific progress in QA. 4 Challenge Problems One of the principal goals of the OAQA is to collaboratively develop a code-base of QA technologies that generalizes over time and may be effectively applied to a broadening collection of QA problems. This goal is driven by the concern that historically, QA systems have been built for a single, specific task and often lack clear and repeatable methods for generalization and adaptation to new domains and problems. We believe that developing integrated end-to-end systems from a common collection of reusable component technologies, in order to target an explicitly crafted set of Challenge Problems, would help ensure more general solutions and greater adaptability. 2 7

9 Challenge Problems are applications of QA technologies designed to test and drive these technologies along key performance dimensions, important for a wide range of business applications of QA. Performance Dimensions include measures like Accuracy, Speed, Question Difficulty, Usability, Breadth of Domain etc. We describe a collection of Performance Dimensions below. Performance Dimensions should ultimately be associated with well-defined metrics and evaluations for any given set of challenge problems. Selecting good challenge problems: In addition to performance dimensions and their metrics, useful challenge problems should captivate the imagination of the scientific community and help to persuade the broader community that advances in automatic question answering can lead to significant and positive impact on science, business and society. The Challenge Problem Set Hypothesis: While, in isolation, a solution to a specific challenge problem may not address a real-world application in its entirety, collaboratively developed solutions to a balanced set of challenge problems will lead to a general QA capability that can be effectively adapted to address a range of business problems with predictable and manageable cost. 4.1 Performance Dimensions In this section we propose a set of Performance Dimensions. We do not, in this white paper, attempt to work out precise quantitative metrics for each of these dimensions, but we acknowledge that this work would be an important contribution for the proposed collaboration on the OAQA. N N is the percentage of questions for which the QA system provides a correct answer in the M th rank in the answer list, where M is <= N for any given question set. N is typically 1 in classic QA evaluations. A Challenge Problem whose success metrics required a high degree of accuracy would rank highly on this dimension whereas one that was tolerant to lower accuracy would rank lower. Accuracy is generally important for building user confidence in the QA application and for reducing the amount of time users spend in vetting wrong answers. Accuracy at 1 is particularly important if the QA system were embedded as a subroutine in a larger system. In these cases a human would not be vetting a collection of alterative answers and their contexts or justification. 8

10 Confidence Accuracy Confidence Accuracy is the probability that the QA system is right about being right. This feature of a QA system assumes that the system provides a Confidence Score, for example, a number between 0 and 1 that indicates a self-assessed probability that the answer(s) it is providing is correct. The degree to which a high Confidence Score correlates with right answers gives the probability that the QA system is right about being right its Confidence Accuracy. Since not all answers will be right nor will they be equally justified by the available content, Confidence Accuracy is a critical feature in the general utility of QA applications. It gives the user or program calling the QA system a realistic indicator about the accuracy of its results. This enables the caller to make better and/or faster decisions. In the case of an embedded QA capability where an application is programmatically calling the QA system directly, Accuracy and Confidence Accuracy are critical metrics. As approaches 100%, Confidence Accuracy becomes less important. However, even in the most optimistic settings, QA systems will not operate at 100% accuracy and Confidence Accuracy will be important in improving the utility of the QA system. Good Confidence Accuracy implies that the system is developing accurate internal justifications for its answers. We distinguish internal justifications from external or user justifications. The former may be effective at predicting the correctness of an answer independently of how it would be understood or evaluated by an end user. External or user justifications are ultimately judged by the user as an acceptable explanation for the correctness of an answer. Although we consider the provision of good user justifications to be a feature of system Usability, a QA system may also consider the presence or absence of external justification(s) as a parameter when calculating answer confidence. Broad Domain Broad Domain refers to the breadth of topics for which the QA system can achieve acceptable levels of accuracy. We do not propose here a formal definition for Broad Domain. However, one can imagine questions where the topics cover a wide variety of domains of knowledge ranging from history and geography to science and medicine to literature, pop culture and current events. We also consider a Broad Domain to imply the scope of the domain is not bounded, but rather that it is open. This implies that the types of things that maybe be asked about in a Challenge Problem that ranks highly in this dimension is considered open-ended and not limited to an a priori fixed set of concepts. 9

11 While we do not propose precise metrics for Broad Domain in this white paper, we can imagine that the breadth of a domain may be measured by the number of specific concepts asked about. The openness of the domain may be measured by the rate in which new concepts grow with the size of the question sets. A challenge problem that focuses narrowly on answering questions about a single topic from a single text would rank low on this dimension, a challenge problem that admitted questions on any topic would rank high. Question Difficulty Question Difficulty refers to the complexity of inference required to determine and justify answers from the available content. The complexity of inference may range from simple string matching to complex logical inference that relies on a formalization of the content and a precisely constructed domain theory of axiomatic knowledge. Question difficulty is relative to the dataset used and in most cases cannot be assessed a priori. Moreover, a question that is easy against one dataset may be hard against another one. In general, known-answer extraction should be considered easier than when an answer needs to be derived or assembled from separate pieces of information retrieved from the dataset. Ideally, a QA system could achieve a high level of accuracy for difficult questions over a broad domain, however, practically and in the near term we expect a tradeoff between Broad Domain and Question Difficulty. In other words, challenge problems that require complex reasoning and inference to answer difficult questions will likely be more effective over narrow domains defined by specific, relatively limited volumes of content. This tradeoff is precipitated, in part, by the human knowledge engineering effort required to formalize the domain theory and axiomatic knowledge required to support complex inference. A desirable property of a QA system is that as richer knowledge representations are added, the ability to handle more difficult questions increases without reducing the system s domain breadth. Query Language Complexity Query Language Complexity refers to the ambiguity and structural complexity of the questions in a QA problem. It does not refer to the formal computational complexity or expressivity of the language. Rather, it relates to the difficulty of extracting the intended meaning of the question from its linguistic expression. So for example, challenge problems that require answering ambiguous, grammatically complex natural language questions would rank higher along this dimension. Challenge 10

12 problems that require the user to express queries in an unambiguous formal language like SQL or SPARCL or admit only questions that instantiate pre-determined templates would rank lower on this dimension. A challenge problem that requires answering ill-formed natural language questions or cross-lingual natural language questions would rank higher on this dimension. Solutions that dialog with the user to map from the user s expression to a formal unambiguous internal representation or that ensure a correct interpretation of a complex user query would score higher in this dimension and likely do well in terms of usability as well. Furthermore, a solution that took this approach might also score higher on the accuracy dimension, given that it is more likely to determine the right interpretation of the query and therefore get the right answer(s). In interactive QA contexts, the need for interactive refinement of the information need to improve accuracy must be balanced against a measure of the overall usability of the system. Content Language Complexity Content Language Complexity is similar to Query Language Complexity but varies independently in Challenge Problems. Challenge Problems defined over well-formed formal language representations of content would rank lower on this dimension. Challenge Problems that must deal with natural language blog data or data for example may rank higher on this dimension. Note that there is less opportunity to engage the user in interpreting the content than in interpreting questions, given that the volumes are huge and it is the principal task of the QA system to automatically interpret the content in order to find the most likely answers and their justifications. Speed (Response Time) Speed refers to the time required to answer a question. A Challenge Problem may range from requiring sub-second response time to batch oriented problems that may allow for 10 s of minutes if not hours per question. A Challenge Problem that required very fast response time would rank high along this dimension. Fast response is ultimately a relative measure. For example, it may be judged relative to how long an average user would take to answer a given set of questions with similar levels of accuracy and confidence but without the use of a QA system. Consider the case where a typical user with the equivalent of an off-the-shelf text search engine took an average of 2 minutes per question on some test set and the QA system took on average 20 minutes per question over the same set with similar levels of accuracy. In this case the QA system s response time would be judged poor. If on the other hand, the QA system answered with an average of 2 seconds per question on the same test set 11

13 with the same or better accuracy as the human, then the QA system would be clearly adding some value. The impact of a significant improvement in speed on any particular business application is yet another question. For example, in a batch research-oriented workflow it may not matter, while in a real-time technical support scenario the rapid response time may be the primary deciding factor for the application of the technology. User Interaction/Usability This performance dimension is intended to describe the degree of user interaction required to succeed on the challenge problem. This dimension can include a mixed bag of important metrics. Most business applications will require features like visualizations/explanations of the query, interactive query refinement, alternative answers, answer contexts, user-acceptable justifications etc. Along this dimension, challenge problems might also address managing the overall information-seeking process of hypothesis generation, query formulation, answer/result management, hypothesis testing and prediction generation. A Challenge Problem that only requires question in/answer out would rank lowly on this dimension. A Challenge Problem whose solution required performing a range of interactions with the user would rank highly on this dimension. Additional Performance Dimensions to Consider In the sample challenge set profile presented below, the performance dimensions listed above are considered. We expect that the list of performance dimensions will be subject to further refinement as new dimensions are proposed and different measures for existing dimensions are explored. The following additional measures have already been noted for future consideration: Answer Specificity / Complexity. A challenge problem could be measured based on the specificity and multiplicity of the answers required (document(s), passages(s), sentence(s), word(s)) and/or the complexity of what counts as an acceptable answer (e.g., a cause-and-effect question like What caused the Sudanese civil war? might best be answered by a set of sentences that describe an interlocking set of events). Note that simple questions can require very complex answers. 4.2 Sample Challenge Set Profile A Challenge Set Profile scores a set of challenge problems in terms of the relative degree to which they would test QA solutions along a key set of performance dimensions. A challenge problem s scores are meaningful within the set, since the scores for any one challenge problem are intended to be relative to the scores considered for the other 12

14 problems in the set. These profiles are qualitative and subjective, but none-the-less help to describe and communicate a relative assessment of different Challenge Problems. A research program should develop crisp metrics for each of its challenge problems and plot solution performance with respect to these metrics along the same performance dimensions used to judge and select the challenge problems themselves. The remainder of this section is an exercise in developing a sample Challenge Set Profile to explore the value of assessing different challenges along the performance dimensions we discussed above. For each problem we consider, we assign a score to each performance dimension with a number between 0 and 10. We plot the results on radar graphs to facilitate a quick visual comparison of the scope of the different challenge problems in the set. Additional work would be required under the OAQA program to more completely and formally describe and rationalize a well-balanced challenge problem set. We describe five challenge problems: 1. TREC QA 2. TAC QA 3. Jeopardy! 4. Learning By Reading 5. Sustained Investigation 13

15 4.2.1 TREC QA Content Language Difficulty Query Language Difficulty Question Difficulty Usability Confidence Broad Domain Accuracy TREC Factoid QA Speed (Response Time) Figure 1: TREC QA Challenge Profile In the TREC QA problem, the task is to answer 500 unseen natural questions from a primary source corpus of roughly 1 million news articles in one week s time, where the primary source had been available for possibly years. The QA system does not interact with a user. It provides one or more answers per question (more than 1 for list questions or in some cases a ranked list of 5 top answers). A solution to TREC QA is not prohibited from using other sources but the answer provided must ultimately come from the primary source. 1. Query Language Difficulty: The questions are free form natural language but are fairly simple in their expression suggesting a relatively low score in Query Language Difficulty. 2. Content Language Difficulty: The primary source is English newswire text stored in structured XML documents. So the content language difficulty is medium to low. 3. Question Difficulty: Most if not all questions may be answered directly by the primary corpora and do not require synthesis or complex inference which would suggest a relatively low score in Question Difficulty. 4. Usability: The TREC QA problem is batch oriented. It requires that a solution provide textual answers to each question but does not allow any interaction with a user, so its Usability score is low. 5. Accuracy: Accuracy is the primary metric in the TREC QA problem. Generally speaking, the more answers judged correct, the better the QA system scores 14

16 relative to others in the challenge. The TREC QA problem ranks relatively high in this dimension. 6. Confidence: With the exception of one year s use of the Confidence Weighted Score, the TREC QA problem generally does not require an accurate confidence score for a solution to excel and therefore ranks lower on this dimension. 7. Speed: The TREC QA problem is batch oriented and systems may utilize up to a week s time to answer 500 questions; this problem ranks lower on the Speed dimension. 8. Broad Domain: The TREC QA problem does not in theory limit itself to any topic or domain and as a challenge problem should score highly along this dimension. The caveat here is that in practice, a historical analysis of the past TREC QA question sets might reveal a broad but limited set of concepts. A formal analysis with more precise metrics might better assess the actual domain breadth of past TREC QA problem instances TAC QA Content Language Difficulty Query Language Difficulty Question Difficulty Usability Confidence Broad Domain Accuracy Speed (Response Time) TREC Factoid QA TAC QA Figure 2: TREC QA and TAC QA As we currently understand the TAC QA (http://www.nist.gov/tac/tracks/2008/qa/) problem (to be evaluated for the first time in the context of TREC 2008), the task is to answer 500 unseen natural questions from a primary source corpus of roughly 3.2 million permalink articles in one week s time, where the primary source has been available since Illustrated in Figure 2, TAC will differ from TREC primarily in the difficulty of the questions, which will include asking about lists of sentiments or opinions, and in the 15

17 content language difficulty which will be directed toward blog posts rather than newswire text. As in the TREC problem, the QA system does not interact with a user. It provides one or more answers per question (more for list questions for example). It is allowed to consider other sources but the answer must ultimately be shown to come from the primary source. 1. Query Language Difficulty: The questions are fairly simple in their expression which would lead to a relatively low score in Query Language Difficulty. 2. Content Language Difficulty: TAC is significantly more difficult than TREC in terms of Content Language Difficulty. The source documents are threaded blog posts, which have more complex structure than newswire texts. The content of blog posts is also written in colloquial language, and is often telegraphic and/or ungrammatical. As a result, TAC ranks more highly on this dimension. 3. Question Difficulty: The questions in TAC Challenge Problem will be more complex than those in TREC QA. They will require identifying spans of text that represent opinions, opinions targets, and opinion holders; question may ask for lists of opinions, targets or holders as well. As a result, TAC ranks more highly on this dimension than TREC. 4. Usability: The TAC QA Problem is batch oriented. It requires that a solution provide textual answers to each question but does not allow any interaction with a user, so its Usability score is low. 5. Accuracy: Accuracy is the primary metric in the TAC QA problem. The more answers judged correct the better the QA system ranks relative to others in the challenge. The TAC QA problem ranks relatively highly in this dimension. 6. Confidence: Like TREC, the TAC QA problem generally does not seem to require an accurate confidence score for a solution to excel and ranks lower on this dimension. 7. Speed: The TAC QA problem is batch oriented and systems may utilize up to a week s time to answer 500 questions; this problem ranks lower on the Speed dimension. 8. Broad Domain: The TAC QA problem does not in theory limit itself to any topic or domain and as a challenge problem should score highly along this dimension. The caveat here is that in practice, an analysis of the corpus might reveal a broad but limited set of concepts that are likely to be the focus of test questions (e.g. consumer products and services). 16

18 4.2.3 Jeopardy! Content Language Difficulty Query Language Difficulty Question Difficulty Usability Confidence Broad Domain Accuracy Speed (Response Time) TREC Factoid QA Jeopardy! Figure 3: Jeopardy! and TREC QA In the Jeopardy! Challenge Problem, the Jeopardy! quiz show is used as a model for integrating a set of metrics into a single QA challenge problem. The task may be simply stated as winning a series of Jeopardy!-style games against a real or simulated champion human player. Succeeding on this challenge requires a combination of high degrees of accuracy, speed and confidence over a broad domain. Moreover, Jeopardy! questions often contain additional details beyond the core question. These details may help find the answer, or they might enhance the entertainment or educational value of the question but not lead directly to the answer. Distinguishing the core question from the tangential details is a hard challenge for an automatic QA system, and requires sophistication in natural language parsing. Jeopardy! provides a QA evaluation framework for additional dimensions of performance and their associated metrics. The game dynamics require high degrees of confidence since the final metric (i.e., winning the game) requires that the system avoids losing questions it chooses to answer, since there is a stiff penalty for getting a question wrong. There is also a requirement to answer interactively and in less than 3-5 seconds. Very high levels of performance will require considering the answers of other players and adjusting confidence levels and follow-up answers according. This may be considered a primitive form of dialog or collaboration. By varying the style and domain of the questions, the same framework could be used to evaluate collaborative and/or competitive question answering in non-jeopardy! domains. In addition, The Jeopardy! challenge has significant potential to capture the imagination and interest of a broader community because of the long-standing public popularity of the 17

19 Jeopardy! quiz show in the U.S. and its general perception as a challenging test of human intelligence (or at least breadth of knowledge). Reaching a larger audience, this challenge problem has the potential to stimulate a broader dialog on automatic Question Answering and a deeper appreciation for its potential. In comparison to the TREC QA problem, the Jeopardy! problem broadens the challenge along key dimensions as shown in the radar graph in Figure Query Language Difficulty: The questions in the Jeopardy! challenge problem are generally more complex than those in TREC QA. They are longer and contain richer language demanding more of parsing technology. They contain multiple cues about the answer as well as tangential information demanding deeper semantic analysis to distinguish reinforcing predicates from irrelevant information. 2. Content Language Difficulty: Jeopardy! ranks similar to or a bit higher than TREC QA on this scale as the necessary sources to achieve higher degrees of accuracy and justification/confidence are broader and more varied. These include general web, news, dictionaries, encyclopedias and hosts of other sources. 3. Question Difficulty: The Jeopardy! Challenge is similar to the TREC QA challenge with respect to question difficulty. There are some questions that require deeper reasoning, and there are a small percentage of list questions. The majority of questions are factoid these are questions that may be answered by one or more entities that satisfy the constraints expressed in the natural language question. 4. Usability: Jeopardy! ranks a bit higher than TREC QA on this dimension. The interaction is largely confined to question in and answer out; it is however interactive rather than batch, and reacting to other players answers is required for high-levels of performance. Also, it should be noted that because the challenge requires a high degree of Confidence, the QA systems generation and use of some sort of justification is implied. This is internal justification, however, and its suitability for consumption by the user is not directly tested by the challenge. This is in contrast to the LBR challenge problem (see below) which requires, for example, humans to judge explanations well in order to succeed. 5. Accuracy: Accuracy is a primary metric for success in this challenge as it is in TREC QA. It focuses on since at most the system will have three chances to deliver the correct answer. Accuracy at 1 is strongly favored. 6. Confidence: Jeopardy! requires a very high degree of confidence, higher than TREC QA or the other challenge problems in the set since the final metric (winning the game) requires that you do not lose on questions you choose to answer. The QA system must compute an accurate confidence in less than 3-5 seconds. 7. Speed: Speed (interactive question answering response time) is critical in this challenge problem. The system must answer in less than 5 seconds. So this problem set ranks highly on this dimension. 8. Broad Domain: The domain in Jeopardy! is arguably broader than TREC QA, requiring access a larger variety of knowledge that goes beyond what might be 18

20 found in a collection of 1 million news articles. We ranked it higher on this dimension Learning by Reading Question Difficulty 10 Content Language Difficulty Query Language Difficulty Usability Confidence Broad Domain Accuracy Speed (Response Time) TREC Factoid QA LBR Jeopardy! Figure 4: LBR, TREC QA and Jeopardy! The Learning by Reading (LBR) challenge problem discussed here is inspired and based on our experience with HALO and the LBR pilot DARPA project. A formal LBR program is currently under development by DARPA. We do not intend to represent what the formal DARPA LBR program may become but rather to use our experience with the HALO and the LBR pilot to propose another interesting and useful QA challenge problem. In the LBR challenge problem, a subject matter text book (i.e., the target text) is selected. This may be, for example, an in depth text on Chemistry, Biology or an even more specific subject like Cardiology. The QA system is built to answer complex questions about the material in the text and must provide answers as well as human understandable justifications. This challenge problem addresses two key dimensions that are weakly addressed by the other problems discussed so far, namely Question Difficulty and Usability. Note that the intended goal of LBR is to demonstrate a system can learn by reading. However, this is demonstrated by an LBR solution s overt behavior to answer users questions. LBR solutions must answer questions that require inference over the logical content of the target text. It is expected that answers can not simply be derived by linguistic inference, but rather would require some domain theory and the ability to map 19

IBM Announces Eight Universities Contributing to the Watson Computing System's Development

IBM Announces Eight Universities Contributing to the Watson Computing System's Development IBM Announces Eight Universities Contributing to the Watson Computing System's Development Press release Related XML feeds Contact(s) information Related resources ARMONK, N.Y. - 11 Feb 2011: IBM (NYSE:

More information

Interoperability, Standards and Open Advancement

Interoperability, Standards and Open Advancement Interoperability, Standards and Open Eric Nyberg 1 Open Shared resources & annotation schemas Shared component APIs Shared datasets (corpora, test sets) Shared software (open source) Shared configurations

More information

Scalable End-User Access to Big Data http://www.optique-project.eu/ HELLENIC REPUBLIC National and Kapodistrian University of Athens

Scalable End-User Access to Big Data http://www.optique-project.eu/ HELLENIC REPUBLIC National and Kapodistrian University of Athens Scalable End-User Access to Big Data http://www.optique-project.eu/ HELLENIC REPUBLIC National and Kapodistrian University of Athens 1 Optique: Improving the competitiveness of European industry For many

More information

IT Workload Automation: Control Big Data Management Costs with Cisco Tidal Enterprise Scheduler

IT Workload Automation: Control Big Data Management Costs with Cisco Tidal Enterprise Scheduler White Paper IT Workload Automation: Control Big Data Management Costs with Cisco Tidal Enterprise Scheduler What You Will Learn Big data environments are pushing the performance limits of business processing

More information

The Prolog Interface to the Unstructured Information Management Architecture

The Prolog Interface to the Unstructured Information Management Architecture The Prolog Interface to the Unstructured Information Management Architecture Paul Fodor 1, Adam Lally 2, David Ferrucci 2 1 Stony Brook University, Stony Brook, NY 11794, USA, pfodor@cs.sunysb.edu 2 IBM

More information

Sentiment Analysis on Big Data

Sentiment Analysis on Big Data SPAN White Paper!? Sentiment Analysis on Big Data Machine Learning Approach Several sources on the web provide deep insight about people s opinions on the products and services of various companies. Social

More information

IBM AND NEXT GENERATION ARCHITECTURE FOR BIG DATA & ANALYTICS!

IBM AND NEXT GENERATION ARCHITECTURE FOR BIG DATA & ANALYTICS! The Bloor Group IBM AND NEXT GENERATION ARCHITECTURE FOR BIG DATA & ANALYTICS VENDOR PROFILE The IBM Big Data Landscape IBM can legitimately claim to have been involved in Big Data and to have a much broader

More information

POLAR IT SERVICES. Business Intelligence Project Methodology

POLAR IT SERVICES. Business Intelligence Project Methodology POLAR IT SERVICES Business Intelligence Project Methodology Table of Contents 1. Overview... 2 2. Visualize... 3 3. Planning and Architecture... 4 3.1 Define Requirements... 4 3.1.1 Define Attributes...

More information

Search and Information Retrieval

Search and Information Retrieval Search and Information Retrieval Search on the Web 1 is a daily activity for many people throughout the world Search and communication are most popular uses of the computer Applications involving search

More information

Streamlining the Process of Business Intelligence with JReport

Streamlining the Process of Business Intelligence with JReport Streamlining the Process of Business Intelligence with JReport An ENTERPRISE MANAGEMENT ASSOCIATES (EMA ) Product Summary from 2014 EMA Radar for Business Intelligence Platforms for Mid-Sized Organizations

More information

The Future of Business Analytics is Now! 2013 IBM Corporation

The Future of Business Analytics is Now! 2013 IBM Corporation The Future of Business Analytics is Now! 1 The pressures on organizations are at a point where analytics has evolved from a business initiative to a BUSINESS IMPERATIVE More organization are using analytics

More information

Toward a Question Answering Roadmap

Toward a Question Answering Roadmap Toward a Question Answering Roadmap Mark T. Maybury 1 The MITRE Corporation 202 Burlington Road Bedford, MA 01730 maybury@mitre.org Abstract Growth in government investment, academic research, and commercial

More information

Find the signal in the noise

Find the signal in the noise Find the signal in the noise Electronic Health Records: The challenge The adoption of Electronic Health Records (EHRs) in the USA is rapidly increasing, due to the Health Information Technology and Clinical

More information

Software Architecture

Software Architecture Cairo University Faculty of Computers and Information Computer Science Department Premasters Studies Software Architecture Report on Software Product Line Submitted to: Dr. Hany Ammar Submitted by: Hadeel

More information

Big Data: Rethinking Text Visualization

Big Data: Rethinking Text Visualization Big Data: Rethinking Text Visualization Dr. Anton Heijs anton.heijs@treparel.com Treparel April 8, 2013 Abstract In this white paper we discuss text visualization approaches and how these are important

More information

So today we shall continue our discussion on the search engines and web crawlers. (Refer Slide Time: 01:02)

So today we shall continue our discussion on the search engines and web crawlers. (Refer Slide Time: 01:02) Internet Technology Prof. Indranil Sengupta Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture No #39 Search Engines and Web Crawler :: Part 2 So today we

More information

(Refer Slide Time: 01:52)

(Refer Slide Time: 01:52) Software Engineering Prof. N. L. Sarda Computer Science & Engineering Indian Institute of Technology, Bombay Lecture - 2 Introduction to Software Engineering Challenges, Process Models etc (Part 2) This

More information

Survey Results: Requirements and Use Cases for Linguistic Linked Data

Survey Results: Requirements and Use Cases for Linguistic Linked Data Survey Results: Requirements and Use Cases for Linguistic Linked Data 1 Introduction This survey was conducted by the FP7 Project LIDER (http://www.lider-project.eu/) as input into the W3C Community Group

More information

BIG DATA & ANALYTICS. Transforming the business and driving revenue through big data and analytics

BIG DATA & ANALYTICS. Transforming the business and driving revenue through big data and analytics BIG DATA & ANALYTICS Transforming the business and driving revenue through big data and analytics Collection, storage and extraction of business value from data generated from a variety of sources are

More information

ANALYTICS STRATEGY: creating a roadmap for success

ANALYTICS STRATEGY: creating a roadmap for success ANALYTICS STRATEGY: creating a roadmap for success Companies in the capital and commodity markets are looking at analytics for opportunities to improve revenue and cost savings. Yet, many firms are struggling

More information

WHY DO I NEED A PROGRAM MANAGEMENT OFFICE (AND HOW DO I GET ONE)?

WHY DO I NEED A PROGRAM MANAGEMENT OFFICE (AND HOW DO I GET ONE)? WHY DO I NEED A PROGRAM MANAGEMENT OFFICE (AND HOW DO I GET ONE)? Due to the often complex and risky nature of projects, many organizations experience pressure for consistency in strategy, communication,

More information

Oracle Real Time Decisions

Oracle Real Time Decisions A Product Review James Taylor CEO CONTENTS Introducing Decision Management Systems Oracle Real Time Decisions Product Architecture Key Features Availability Conclusion Oracle Real Time Decisions (RTD)

More information

Integrated Social and Enterprise Data = Enhanced Analytics

Integrated Social and Enterprise Data = Enhanced Analytics ORACLE WHITE PAPER, DECEMBER 2013 THE VALUE OF SOCIAL DATA Integrated Social and Enterprise Data = Enhanced Analytics #SocData CONTENTS Executive Summary 3 The Value of Enterprise-Specific Social Data

More information

Apache Hadoop: The Big Data Refinery

Apache Hadoop: The Big Data Refinery Architecting the Future of Big Data Whitepaper Apache Hadoop: The Big Data Refinery Introduction Big data has become an extremely popular term, due to the well-documented explosion in the amount of data

More information

Text Analytics with Ambiverse. Text to Knowledge. www.ambiverse.com

Text Analytics with Ambiverse. Text to Knowledge. www.ambiverse.com Text Analytics with Ambiverse Text to Knowledge www.ambiverse.com Version 1.0, February 2016 WWW.AMBIVERSE.COM Contents 1 Ambiverse: Text to Knowledge............................... 5 1.1 Text is all Around

More information

Data Migration through an Information Development Approach An Executive Overview

Data Migration through an Information Development Approach An Executive Overview Data Migration through an Approach An Executive Overview Introducing MIKE2.0 An Open Source Methodology for http://www.openmethodology.org Management and Technology Consultants Data Migration through an

More information

CHAPTER 1 INTRODUCTION

CHAPTER 1 INTRODUCTION 1 CHAPTER 1 INTRODUCTION Exploration is a process of discovery. In the database exploration process, an analyst executes a sequence of transformations over a collection of data structures to discover useful

More information

A GENERAL TAXONOMY FOR VISUALIZATION OF PREDICTIVE SOCIAL MEDIA ANALYTICS

A GENERAL TAXONOMY FOR VISUALIZATION OF PREDICTIVE SOCIAL MEDIA ANALYTICS A GENERAL TAXONOMY FOR VISUALIZATION OF PREDICTIVE SOCIAL MEDIA ANALYTICS Stacey Franklin Jones, D.Sc. ProTech Global Solutions Annapolis, MD Abstract The use of Social Media as a resource to characterize

More information

Information Visualization WS 2013/14 11 Visual Analytics

Information Visualization WS 2013/14 11 Visual Analytics 1 11.1 Definitions and Motivation Lot of research and papers in this emerging field: Visual Analytics: Scope and Challenges of Keim et al. Illuminating the path of Thomas and Cook 2 11.1 Definitions and

More information

Realestate online information systems

Realestate online information systems Realestate online information systems Yuri Martens, Alexander Koutamanis Faculty of Architecture, Delft University of Technology http://www.re-h.nl Abstract. Several commercial real-estate sites provide

More information

IBM WebSphere Business Monitor, Version 6.1

IBM WebSphere Business Monitor, Version 6.1 Providing real-time visibility into business performance IBM, Version 6.1 Highlights Enables business users to view Integrates with IBM s BPM near real-time data on Web 2.0 portfolio and non-ibm dashboards

More information

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material,

More information

Visualization methods for patent data

Visualization methods for patent data Visualization methods for patent data Treparel 2013 Dr. Anton Heijs (CTO & Founder) Delft, The Netherlands Introduction Treparel can provide advanced visualizations for patent data. This document describes

More information

Implement a unified approach to service quality management.

Implement a unified approach to service quality management. Service quality management solutions To support your business objectives Implement a unified approach to service quality management. Highlights Deliver high-quality software applications that meet functional

More information

Better planning and forecasting with IBM Predictive Analytics

Better planning and forecasting with IBM Predictive Analytics IBM Software Business Analytics SPSS Predictive Analytics Better planning and forecasting with IBM Predictive Analytics Using IBM Cognos TM1 with IBM SPSS Predictive Analytics to build better plans and

More information

A discussion of information integration solutions November 2005. Deploying a Center of Excellence for data integration.

A discussion of information integration solutions November 2005. Deploying a Center of Excellence for data integration. A discussion of information integration solutions November 2005 Deploying a Center of Excellence for data integration. Page 1 Contents Summary This paper describes: 1 Summary 1 Introduction 2 Mastering

More information

A Hurwitz white paper. Inventing the Future. Judith Hurwitz President and CEO. Sponsored by Hitachi

A Hurwitz white paper. Inventing the Future. Judith Hurwitz President and CEO. Sponsored by Hitachi Judith Hurwitz President and CEO Sponsored by Hitachi Introduction Only a few years ago, the greatest concern for businesses was being able to link traditional IT with the requirements of business units.

More information

Extend the value of your core business systems.

Extend the value of your core business systems. Legacy systems renovation to SOA September 2006 Extend the value of your core business systems. Transforming legacy applications into an SOA framework Page 2 Contents 2 Unshackling your core business systems

More information

Client Onboarding Process Reengineering: Performance Management of Client Onboarding Programs

Client Onboarding Process Reengineering: Performance Management of Client Onboarding Programs KNOWLEDGENT INSIGHTS volume 1 no. 4 September 13, 2011 Client Onboarding Process Reengineering: Performance Management of Client Onboarding Programs In the midst of the worst economic environment since

More information

Standardized Technology Evaluation Process (STEP) User s Guide and Methodology for Evaluation Teams. Sarah Brown May 2007

Standardized Technology Evaluation Process (STEP) User s Guide and Methodology for Evaluation Teams. Sarah Brown May 2007 Standardized Technology Evaluation Process (STEP) User s Guide and Methodology for Evaluation Teams Sarah Brown May 2007 1 Table of Contents 1 Introduction 4 1.1 Purpose 4 1.2 Background 5 1.3 Intended

More information

Managing Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database

Managing Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Managing Big Data with Hadoop & Vertica A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Copyright Vertica Systems, Inc. October 2009 Cloudera and Vertica

More information

Health Data Analytics. Data to Value For Small and Medium Healthcare organizations

Health Data Analytics. Data to Value For Small and Medium Healthcare organizations Health Data Analytics Data to Value For Small and Medium Healthcare organizations HEALTH DATA ANALYTICS WHITE PAPER JULY 2013 GREENCASTLE CONSULTING Abstract This paper is targeted toward small and medium

More information

Next-Generation Building Energy Management Systems

Next-Generation Building Energy Management Systems WHITE PAPER Next-Generation Building Energy Management Systems New Opportunities and Experiences Enabled by Intelligent Equipment Published 2Q 2015 Sponsored By Daikin Applied and Intel Casey Talon Senior

More information

OPTIMUS SBR. Optimizing Results with Business Intelligence Governance CHOICE TOOLS. PRECISION AIM. BOLD ATTITUDE.

OPTIMUS SBR. Optimizing Results with Business Intelligence Governance CHOICE TOOLS. PRECISION AIM. BOLD ATTITUDE. OPTIMUS SBR CHOICE TOOLS. PRECISION AIM. BOLD ATTITUDE. Optimizing Results with Business Intelligence Governance This paper investigates the importance of establishing a robust Business Intelligence (BI)

More information

SUSTAINING COMPETITIVE DIFFERENTIATION

SUSTAINING COMPETITIVE DIFFERENTIATION SUSTAINING COMPETITIVE DIFFERENTIATION Maintaining a competitive edge in customer experience requires proactive vigilance and the ability to take quick, effective, and unified action E M C P e r s pec

More information

Keywords Big Data; OODBMS; RDBMS; hadoop; EDM; learning analytics, data abundance.

Keywords Big Data; OODBMS; RDBMS; hadoop; EDM; learning analytics, data abundance. Volume 4, Issue 11, November 2014 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Analytics

More information

III Big Data Technologies

III Big Data Technologies III Big Data Technologies Today, new technologies make it possible to realize value from Big Data. Big data technologies can replace highly customized, expensive legacy systems with a standard solution

More information

Integrating SAP and non-sap data for comprehensive Business Intelligence

Integrating SAP and non-sap data for comprehensive Business Intelligence WHITE PAPER Integrating SAP and non-sap data for comprehensive Business Intelligence www.barc.de/en Business Application Research Center 2 Integrating SAP and non-sap data Authors Timm Grosser Senior Analyst

More information

Text Mining - Scope and Applications

Text Mining - Scope and Applications Journal of Computer Science and Applications. ISSN 2231-1270 Volume 5, Number 2 (2013), pp. 51-55 International Research Publication House http://www.irphouse.com Text Mining - Scope and Applications Miss

More information

Data Modeling for Big Data

Data Modeling for Big Data Data Modeling for Big Data by Jinbao Zhu, Principal Software Engineer, and Allen Wang, Manager, Software Engineering, CA Technologies In the Internet era, the volume of data we deal with has grown to terabytes

More information

BSC vision on Big Data and extreme scale computing

BSC vision on Big Data and extreme scale computing BSC vision on Big Data and extreme scale computing Jesus Labarta, Eduard Ayguade,, Fabrizio Gagliardi, Rosa M. Badia, Toni Cortes, Jordi Torres, Adrian Cristal, Osman Unsal, David Carrera, Yolanda Becerra,

More information

» A Hardware & Software Overview. Eli M. Dow

» A Hardware & Software Overview. Eli M. Dow <emdow@us.ibm.com:> » A Hardware & Software Overview Eli M. Dow Overview:» Hardware» Software» Questions 2011 IBM Corporation Early implementations of Watson ran on a single processor where it took 2 hours

More information

Solve your toughest challenges with data mining

Solve your toughest challenges with data mining IBM Software IBM SPSS Modeler Solve your toughest challenges with data mining Use predictive intelligence to make good decisions faster Solve your toughest challenges with data mining Imagine if you could

More information

Big Data and Data Analytics

Big Data and Data Analytics 2.0 Big Data and Data Analytics (Volume 18, Number 3) By Heather A. Smith James D. McKeen Sponsored by: Introduction At a time when organizations are just beginning to do the hard work of standardizing

More information

Data Refinery with Big Data Aspects

Data Refinery with Big Data Aspects International Journal of Information and Computation Technology. ISSN 0974-2239 Volume 3, Number 7 (2013), pp. 655-662 International Research Publications House http://www. irphouse.com /ijict.htm Data

More information

Delivering Customer Value Faster With Big Data Analytics

Delivering Customer Value Faster With Big Data Analytics Delivering Customer Value Faster With Big Data Analytics Tackle the challenges of Big Data and real-time analytics with a cloud-based Decision Management Ecosystem James Taylor CEO Customer data is more

More information

Advanced Big Data Analytics with R and Hadoop

Advanced Big Data Analytics with R and Hadoop REVOLUTION ANALYTICS WHITE PAPER Advanced Big Data Analytics with R and Hadoop 'Big Data' Analytics as a Competitive Advantage Big Analytics delivers competitive advantage in two ways compared to the traditional

More information

Understanding the Value of In-Memory in the IT Landscape

Understanding the Value of In-Memory in the IT Landscape February 2012 Understing the Value of In-Memory in Sponsored by QlikView Contents The Many Faces of In-Memory 1 The Meaning of In-Memory 2 The Data Analysis Value Chain Your Goals 3 Mapping Vendors to

More information

Navigating Big Data business analytics

Navigating Big Data business analytics mwd a d v i s o r s Navigating Big Data business analytics Helena Schwenk A special report prepared for Actuate May 2013 This report is the third in a series and focuses principally on explaining what

More information

Big Data Services From Hitachi Data Systems

Big Data Services From Hitachi Data Systems SOLUTION PROFILE Big Data Services From Hitachi Data Systems Create Strategy, Implement and Manage a Solution for Big Data for Your Organization Big Data Consulting Services and Big Data Transition Services

More information

OPEN MODERN DATA ARCHITECTURE FOR FINANCIAL SERVICES RISK MANAGEMENT

OPEN MODERN DATA ARCHITECTURE FOR FINANCIAL SERVICES RISK MANAGEMENT WHITEPAPER OPEN MODERN DATA ARCHITECTURE FOR FINANCIAL SERVICES RISK MANAGEMENT A top-tier global bank s end-of-day risk analysis jobs didn t complete in time for the next start of trading day. To solve

More information

WHITE PAPER. Harnessing the Power of Advanced Analytics How an appliance approach simplifies the use of advanced analytics

WHITE PAPER. Harnessing the Power of Advanced Analytics How an appliance approach simplifies the use of advanced analytics WHITE PAPER Harnessing the Power of Advanced How an appliance approach simplifies the use of advanced analytics Introduction The Netezza TwinFin i-class advanced analytics appliance pushes the limits of

More information

[project.headway] Integrating Project HEADWAY And CMMI

[project.headway] Integrating Project HEADWAY And CMMI [project.headway] I N T E G R A T I O N S E R I E S Integrating Project HEADWAY And CMMI P R O J E C T H E A D W A Y W H I T E P A P E R Integrating Project HEADWAY And CMMI Introduction This white paper

More information

In-Database Analytics

In-Database Analytics Embedding Analytics in Decision Management Systems In-database analytics offer a powerful tool for embedding advanced analytics in a critical component of IT infrastructure. James Taylor CEO CONTENTS Introducing

More information

Interactive data analytics drive insights

Interactive data analytics drive insights Big data Interactive data analytics drive insights Daniel Davis/Invodo/S&P. Screen images courtesy of Landmark Software and Services By Armando Acosta and Joey Jablonski The Apache Hadoop Big data has

More information

Increasing Development Knowledge with EPFC

Increasing Development Knowledge with EPFC The Eclipse Process Framework Composer Increasing Development Knowledge with EPFC Are all your developers on the same page? Are they all using the best practices and the same best practices for agile,

More information

CONFIOUS * : Managing the Electronic Submission and Reviewing Process of Scientific Conferences

CONFIOUS * : Managing the Electronic Submission and Reviewing Process of Scientific Conferences CONFIOUS * : Managing the Electronic Submission and Reviewing Process of Scientific Conferences Manos Papagelis 1, 2, Dimitris Plexousakis 1, 2 and Panagiotis N. Nikolaou 2 1 Institute of Computer Science,

More information

Turning Big Data into Big Insights

Turning Big Data into Big Insights mwd a d v i s o r s Turning Big Data into Big Insights Helena Schwenk A special report prepared for Actuate May 2013 This report is the fourth in a series and focuses principally on explaining what s needed

More information

Introduction to SOA governance and service lifecycle management.

Introduction to SOA governance and service lifecycle management. -oriented architecture White paper March 2009 Introduction to SOA governance and Best practices for development and deployment Bill Brown, executive IT architect, worldwide SOA governance SGMM lead, SOA

More information

Putting IBM Watson to Work In Healthcare

Putting IBM Watson to Work In Healthcare Martin S. Kohn, MD, MS, FACEP, FACPE Chief Medical Scientist, Care Delivery Systems IBM Research marty.kohn@us.ibm.com Putting IBM Watson to Work In Healthcare 2 SB 1275 Medical data in an electronic or

More information

DESIGN AND STRUCTURE OF FUZZY LOGIC USING ADAPTIVE ONLINE LEARNING SYSTEMS

DESIGN AND STRUCTURE OF FUZZY LOGIC USING ADAPTIVE ONLINE LEARNING SYSTEMS Abstract: Fuzzy logic has rapidly become one of the most successful of today s technologies for developing sophisticated control systems. The reason for which is very simple. Fuzzy logic addresses such

More information

Cray: Enabling Real-Time Discovery in Big Data

Cray: Enabling Real-Time Discovery in Big Data Cray: Enabling Real-Time Discovery in Big Data Discovery is the process of gaining valuable insights into the world around us by recognizing previously unknown relationships between occurrences, objects

More information

Semantic Web in 10 years: Semantics with a purpose

Semantic Web in 10 years: Semantics with a purpose Semantic Web in 10 years: Semantics with a purpose [Position paper for the ISWC2012 Workshop on What will the Semantic Web look like 10 years from now? ] Marko Grobelnik, Dunja Mladenić, Blaž Fortuna J.

More information

White Paper. How Streaming Data Analytics Enables Real-Time Decisions

White Paper. How Streaming Data Analytics Enables Real-Time Decisions White Paper How Streaming Data Analytics Enables Real-Time Decisions Contents Introduction... 1 What Is Streaming Analytics?... 1 How Does SAS Event Stream Processing Work?... 2 Overview...2 Event Stream

More information

www.pwc.com Internal Audit Data Analytics

www.pwc.com Internal Audit Data Analytics www.pwc.com Internal Audit Data Analytics What s driving the need for enhanced data analytics in the market Analytics maturity model Capabilities developed and adopted Capabilities used to drive audits

More information

white paper Measure. Manage. Improve: Unlocking the Business Value of Software Development Optimization

white paper Measure. Manage. Improve: Unlocking the Business Value of Software Development Optimization white paper Measure. Manage. Improve: Unlocking the Business Value of Software Development Optimization EXECUTIVE SUMMARY In 2011 the Standish Group s CHAOS Manifesto showed that 37% of software projects

More information

Business Process Management In An Application Development Environment

Business Process Management In An Application Development Environment Business Process Management In An Application Development Environment Overview Today, many core business processes are embedded within applications, such that it s no longer possible to make changes to

More information

Basel Committee on Banking Supervision. Working Paper No. 17

Basel Committee on Banking Supervision. Working Paper No. 17 Basel Committee on Banking Supervision Working Paper No. 17 Vendor models for credit risk measurement and management Observations from a review of selected models February 2010 The Working Papers of the

More information

Business Architecture Guild Body of Knowledge Handbook 2.0

Business Architecture Guild Body of Knowledge Handbook 2.0 Guild Body of Knowledge Handbook 2.0 ------------------------ Section 1: Introduction The Guild has made this Introduction section of its Body of Knowledge Handbook 2.0 ( Handbook ) publicly available

More information

High-Performance Business Analytics: SAS and IBM Netezza Data Warehouse Appliances

High-Performance Business Analytics: SAS and IBM Netezza Data Warehouse Appliances High-Performance Business Analytics: SAS and IBM Netezza Data Warehouse Appliances Highlights IBM Netezza and SAS together provide appliances and analytic software solutions that help organizations improve

More information

IBM Software IBM Business Process Management Suite. Increase business agility with the IBM Business Process Management Suite

IBM Software IBM Business Process Management Suite. Increase business agility with the IBM Business Process Management Suite IBM Software IBM Business Process Management Suite Increase business agility with the IBM Business Process Management Suite 2 Increase business agility with the IBM Business Process Management Suite We

More information

Business Intelligence and Big Data Analytics: Speeding the Cycle from Insights to Action Four Steps to More Profitable Customer Engagement

Business Intelligence and Big Data Analytics: Speeding the Cycle from Insights to Action Four Steps to More Profitable Customer Engagement white paper Business Intelligence and Big Data Analytics: Speeding the Cycle from Insights to Action Four Steps to More Profitable Customer Engagement»» Summary For business intelligence analysts the era

More information

IBM Content Analytics adds value to Cognos BI

IBM Content Analytics adds value to Cognos BI IBM Software IBM Industry Solutions IBM Content Analytics adds value to Cognos BI 2 IBM Content Analytics adds value to Cognos BI Analyzing unstructured information It is generally accepted that about

More information

Cisco Data Preparation

Cisco Data Preparation Data Sheet Cisco Data Preparation Unleash your business analysts to develop the insights that drive better business outcomes, sooner, from all your data. As self-service business intelligence (BI) and

More information

Chartis RiskTech Quadrant for Model Risk Management Systems 2014

Chartis RiskTech Quadrant for Model Risk Management Systems 2014 Chartis RiskTech Quadrant for Model Risk Management Systems 2014 The RiskTech Quadrant is copyrighted June 2014 by Chartis Research Ltd. and is reused with permission. No part of the RiskTech Quadrant

More information

HOW TO DO A SMART DATA PROJECT

HOW TO DO A SMART DATA PROJECT April 2014 Smart Data Strategies HOW TO DO A SMART DATA PROJECT Guideline www.altiliagroup.com Summary ALTILIA s approach to Smart Data PROJECTS 3 1. BUSINESS USE CASE DEFINITION 4 2. PROJECT PLANNING

More information

Telecommunication (120 ЕCTS)

Telecommunication (120 ЕCTS) Study program Faculty Cycle Software Engineering and Telecommunication (120 ЕCTS) Contemporary Sciences and Technologies Postgraduate ECTS 120 Offered in Tetovo Description of the program This master study

More information

Six ways to accelerate Android mobile application development

Six ways to accelerate Android mobile application development Six ways to accelerate Android mobile application Creating an integrated solution for collaboration among teams Contents 1 Weaving the invisible thread of innovation 2 Android : vast opportunities and

More information

Software-assisted document review: An ROI your GC can appreciate. kpmg.com

Software-assisted document review: An ROI your GC can appreciate. kpmg.com Software-assisted document review: An ROI your GC can appreciate kpmg.com b Section or Brochure name Contents Introduction 4 Approach 6 Metrics to compare quality and effectiveness 7 Results 8 Matter 1

More information

Integrating a Big Data Platform into Government:

Integrating a Big Data Platform into Government: Integrating a Big Data Platform into Government: Drive Better Decisions for Policy and Program Outcomes John Haddad, Senior Director Product Marketing, Informatica Digital Government Institute s Government

More information

Solve Your Toughest Challenges with Data Mining

Solve Your Toughest Challenges with Data Mining IBM Software Business Analytics IBM SPSS Modeler Solve Your Toughest Challenges with Data Mining Use predictive intelligence to make good decisions faster Solve Your Toughest Challenges with Data Mining

More information

Symantec Global Intelligence Network 2.0 Architecture: Staying Ahead of the Evolving Threat Landscape

Symantec Global Intelligence Network 2.0 Architecture: Staying Ahead of the Evolving Threat Landscape WHITE PAPER: SYMANTEC GLOBAL INTELLIGENCE NETWORK 2.0.... ARCHITECTURE.................................... Symantec Global Intelligence Network 2.0 Architecture: Staying Ahead of the Evolving Threat Who

More information

IBM Global Business Services Microsoft Dynamics CRM solutions from IBM

IBM Global Business Services Microsoft Dynamics CRM solutions from IBM IBM Global Business Services Microsoft Dynamics CRM solutions from IBM Power your productivity 2 Microsoft Dynamics CRM solutions from IBM Highlights Win more deals by spending more time on selling and

More information

Accelerate BI Initiatives With Self-Service Data Discovery And Integration

Accelerate BI Initiatives With Self-Service Data Discovery And Integration A Custom Technology Adoption Profile Commissioned By Attivio June 2015 Accelerate BI Initiatives With Self-Service Data Discovery And Integration Introduction The rapid advancement of technology has ushered

More information

Andre Standback. IT 103, Sec. 001 2/21/12. IBM s Watson. GMU Honor Code on http://academicintegrity.gmu.edu/honorcode/. I am fully aware of the

Andre Standback. IT 103, Sec. 001 2/21/12. IBM s Watson. GMU Honor Code on http://academicintegrity.gmu.edu/honorcode/. I am fully aware of the Andre Standback IT 103, Sec. 001 2/21/12 IBM s Watson "By placing this statement on my webpage, I certify that I have read and understand the GMU Honor Code on http://academicintegrity.gmu.edu/honorcode/.

More information

Lambda Architecture. Near Real-Time Big Data Analytics Using Hadoop. January 2015. Email: bdg@qburst.com Website: www.qburst.com

Lambda Architecture. Near Real-Time Big Data Analytics Using Hadoop. January 2015. Email: bdg@qburst.com Website: www.qburst.com Lambda Architecture Near Real-Time Big Data Analytics Using Hadoop January 2015 Contents Overview... 3 Lambda Architecture: A Quick Introduction... 4 Batch Layer... 4 Serving Layer... 4 Speed Layer...

More information

Predictive Analytics Certificate Program

Predictive Analytics Certificate Program Information Technologies Programs Predictive Analytics Certificate Program Accelerate Your Career Offered in partnership with: University of California, Irvine Extension s professional certificate and

More information

Data analytics Delivering intelligence in the moment

Data analytics Delivering intelligence in the moment www.pwc.co.uk Data analytics Delivering intelligence in the moment January 2014 Our point of view Extracting insight from an organisation s data and applying it to business decisions has long been a necessary

More information

A Framework for the Delivery of Personalized Adaptive Content

A Framework for the Delivery of Personalized Adaptive Content A Framework for the Delivery of Personalized Adaptive Content Colm Howlin CCKF Limited Dublin, Ireland colm.howlin@cckf-it.com Danny Lynch CCKF Limited Dublin, Ireland colm.howlin@cckf-it.com Abstract

More information