EPL 660: Lab 1 General Info, Exercise 1, B-Trees, Apache Lucene

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "EPL 660: Lab 1 General Info, Exercise 1, B-Trees, Apache Lucene"

Transcription

1 EPL 660: Lab 1 General Info, Exercise 1, B-Trees, Apache Lucene Andreas Kamilaris Department of Computer Science Created by Andreas Kamilaris for EPL660

2 Research on the Web of Things 2

3 General info Every Friday 18:00-19:30. Check course Web site for schedule. Lab content - Exercises, general questions, tutorials, tool demonstrations. Deadlines of exercises: 23:59 at delivery day. submission: EPL660 3

4 Tutorials info Review of tools for Information Retrieval. Every lab session includes introducing some tool. A variety of libraries and tools: Apache Lucene Apache Solr Apache Tika Hadoop Nutch EPL660 4

5 Program info Date 28/01 4/02 11/02 18/02 25/02 4/03 11/03 18/03 25/03 01/04 8/04 15/04 Topic Apache Lucene Apache Lucene Apache Solr Apache Tika No Tutorial Hadoop Hadoop Nutch No Tutorial No Tutorial Nutch Projects Presentations Description Introduction to Apache Lucene Background Information for B-Trees Getting Started with Apache Lucene Demonstration of a simple scenario Introduction to Apache Solr Demonstration of a simple scenario Introduction to Apache Tika Demostration of a simple scenario Absence of Assistant Background information about MapReduce Introduction to Hadoop Getting Started with Hadoop Demonstration of a simple scenario Background Information about Crawling Introduction to Nutch Public Holiday Public Holiday Getting Started with Nutch Presentation of the students final project EPL660 5

6 1 st Programming Exercise Create a doc-based inverted index. Records have the format: term Frequency Positional Posting List Include stemming using Porter Stemmer algorithm. Include detection of stop-words. Search terms using B-Trees. The B-Tree must be a 4-ordered tree. Add skip pointers to inverted index for performance reasons. EPL660 6

7 1 st Programming Exercise Deadline is 8 th February You need to include: Source code with comments. Executable files. A Brief Documentation. Submission including a zip attachment. EPL660 7

8 Introduction to B-Trees A B-Tree of order m is an m-way tree (a tree where each node may have up to m children) in which: 1. the number of keys in each non-leaf node is one less than the number of its children and these keys partition the keys in the children in the fashion of a search tree. 2. all leaves are on the same level. 3. all non-leaf nodes except the root have at least m / 2 children. 4. the root is either a leaf node, or it has from two to m children. 5. a leaf node contains no more than m 1 keys. B-trees are always balanced! EPL660 8

9 Why using B-Trees It was difficult to access a large amount of data from a secondary memory. Many algorithms were introduced to make search faster, to access the required data from the secondary memory more optimized. B-Trees are more effective and faster. B-Trees are used in many database management systems. EPL660 9

10 An example B-Tree A B-tree of order 4 containing 26 items: Note that all the leaves are at the same level EPL660 10

11 Searching a B-Tree Search for the item #48: Note that all the leaves are at the same level EPL660 11

12 Constructing a B-Tree Suppose we start with an empty B-tree and keys arrive in the following order: We want to construct a B-tree of order 5 The first four items go into the root: To put the fifth item in the root would violate condition 5 Therefore, when 25 arrives, pick the middle key to make a new root EPL660 12

13 Constructing a B-Tree , 14, 28 get added to the leaf nodes: EPL660 13

14 Constructing a B-Tree Adding 17 to the right leaf node would over-fill it, so we take the middle key, promote it (to the root) and split the leaf: , 52, 16, 48 get added to the leaf nodes: EPL660 14

15 Constructing a B-Tree Adding 68 causes us to split the right most leaf, promoting 48 to the root, and adding 3 causes us to split the left most leaf, promoting 3 to the root; 26, 29, 53, 55 then go into the leaves: Adding 45 causes a split of: and promoting 28 to the root then causes the root to split. EPL660 15

16 Constructing a B-Tree EPL660 16

17 Guidelines for constructing a B-Tree 1. Attempt to insert the new key into a leaf by searching for the proper position. 2. If the leaf is not full, then insert the key and you are done. 3. If this would result in that leaf becoming too big, split the leaf into two, promoting the middle key to the leaf s parent 4. If this would result in the parent becoming too big, split the parent into two, promoting the middle key. 5. This strategy might have to be repeated all the way to the top. 6. If necessary, the root is split in two and the middle key is promoted to a new root, making the tree one level higher. EPL660 17

18 Time complexity of a B-Tree Search/Insert/Delete all take up to the number of items in a path from the root to a leaf. The total number of operations is no more than the height of the tree. The height of a tree is no more than log(n) where n is the number of items in the B-Tree. EPL660 18

19 Tutorial 1 Apache Lucene Overview Department of Computer Science

20 What is Apache Lucene? Apache Lucene is a high-performance, fullfeatured text search engine library written entirely in Java. -from EPL660 20

21 What is Apache Lucene? Lucene is specifically an API, not an application. Hard parts have been done, easy programming has been left to you. You can build a search application that is specifically suited to your needs. You can use Lucene to provide consistent full-text indexing across both database objects and documents in various formats (Microsoft Office documents, PDF, HTML, text, s and so on). EPL660 21

22 Availability Freely Available (no cost) Open Source Apache License, version Download from: EPL660 22

23 Features Ranked Searching Flexible Queries Phrases, Wildcards, etc Field-specific Queries e.g. title, artist, album Sorting EPL660 23

24 Ranked Searching 1. Phrase Matching 2. Keyword Matching Prefer more unique terms first takes into account the uniqueness of each term when determining a document s relevance score EPL660 24

25 Flexible Queries Phrases star wars Wildcards star* Bra?il Ranges {star-stun} [ ] Boolean Operators star AND wars This is just a small subset of the types of queries that Lucene can support. Some query types such as wildcard and range queries have a potential to cause heavy load on the Lucene server, so Lucene makes it easy to disable certain types of queries while allowing all others to proceed through the system. This gives programmers better control and allows the system performance to be more predictable. EPL660 25

26 Field-specific Queries For example title: star wars AND director: George Lucas EPL660 26

27 Sorting Can sort any field in a Document For example, by Price, Release Date, Amazon Sales Rank, etc By default, Lucene will sort results by their relevance score. Sorting by any other field in a Document is also supported. EPL660 27

28 Documents A document can represent anything textual: Word Document DVD (the textual metadata only) Website Member (name, ID, etc ) A Lucene Document need not refer to an actual file on a disk, it could also resemble a row in a relational database. Each developer is responsible for turning their own data sets into Lucene Documents. Lucene comes with a number of 3rd party contributions, including examples for parsing structured data files such as XML documents and Word files. EPL660 28

29 Indexes Lucene employs inverted indexing (like most full-textbased search engines). Indexes track term frequencies. Every term maps back to a Document. This index is what allows Lucene to quickly locate every document currently associated with a given set of input search terms. EPL660 29

30 Basic Indexing An index consists of one or more Lucene documents. 1. Create a document: A document consists of one or more fields: name-value pair Example: A field commonly found in applications is title. In the case of a title field, the field name is title and the value is the title of that item. Add one or more fields to the document. 2. Add the document to an index: Indexing involves adding documents to an IndexWriter. 3. Indexer will analyze the Document: We can provide specialized analyzers such as StandardAnalyzer. EPL660 30

31 Analyzing Analyzers control how the text is broken into terms which are then used to index the document. Analyzers can be used to remove stop words and they also perform stemming. Lucene comes with a default analyzer which works well for unstructured English text, however it often performs incorrect normalizations on non-english texts. Lucene makes it easy to build custom Analyzers, and provides a number of helpful building blocks with which to build your own. Lucene even includes a number of stemming algorithms for various languages, which can improve document retrieval accuracy when the source language is known at indexing time. EPL660 31

32 Basic Searching Searching requires an index to have already been built. 1. Create a Query: Usually via QueryParser, MultiPhraseQuery etc. that parse user input. 2. Open an Index: 3. Search the Index: E.g. via IndexSearcher. Use an Analyzer (as before). 4. Iterate through returned Documents: Extract out needed results. Extract out result scores (if needed). EPL660 32

33 Lucene as a Web Service 1. Design an HTTP query syntax GET queries XML for results 2. Wrap Tomcat around core code Tomcat is a source software implementation of the Java Servlet and JavaServer Pages technologies 3. Write a Client Library EPL660 33

34 Scalability Limits 3 main scalability factors: Query Rate Index Size Update Rate EPL660 34

35 Query Rate Scalability Lucene is already fast: Built-in simple cache mechanism Easy solution for heavy workloads: Add more query servers behind a load balancer Can grow as your traffic grows EPL660 35

36 Index Size Scalability Can easily handle millions of documents Lucene is very commonly deployed into systems with 10s of millions of documents. Although query performance can degrade as more documents are added to the index, the growth factor is very low. The main limits related to index size that you are likely to run into, will be disk capacity and disk I/O limits. If you need bigger index: Built-in methods to allow queries to span multiple remote Lucene indexes Can merge multiple remote indexes at query-time. EPL660 36

37 Lucene Installation 1. Download the latest version of Lucene (v3.0.3) from: 2. Add files lucene-core-{version}.jar and lucene-demos- {version}.jar in your Java CLASSPATH. 3. Start programming! (Optional Step) 4. Go to Lucene-{version}/src/demo/org/apache/lucene/demo directory and start editing files IndexFiles.java and SearchFiles.java. EPL660 37

38 Useful Info Official Apache Lucene site: Lucene-java Wiki: Lucene Intro (java.net): Lucene Tutorial.com: EPL660 38

Apache Lucene. Searching the Web and Everything Else. Daniel Naber Mindquarry GmbH ID 380

Apache Lucene. Searching the Web and Everything Else. Daniel Naber Mindquarry GmbH ID 380 Apache Lucene Searching the Web and Everything Else Daniel Naber Mindquarry GmbH ID 380 AGENDA 2 > What's a search engine > Lucene Java Features Code example > Solr Features Integration > Nutch Features

More information

Open Source IR Tools and Libraries

Open Source IR Tools and Libraries Open Source IR Tools and Libraries Giorgos Vasiliadis, gvasil@csd.uoc.gr CS-463 Information Retrieval Models Computer Science Department University of Crete 1 Outline Google Search API Lucene Terrier Lemur

More information

CS242 PROJECT. Presented by Moloud Shahbazi Spring 2015

CS242 PROJECT. Presented by Moloud Shahbazi Spring 2015 CS242 PROJECT Presented by Moloud Shahbazi Spring 2015 AGENDA Project Overview Data Collection Indexing Big Data Processing PROJECT- PART1 1.1 Data Collection: 5G < data size < 10G Deliverables: Document

More information

Indexing big data with Tika, Solr, and map-reduce

Indexing big data with Tika, Solr, and map-reduce Indexing big data with Tika, Solr, and map-reduce Scott Fisher, Erik Hetzner California Digital Library 8 February 2012 Scott Fisher, Erik Hetzner (CDL) Indexing big data 8 February 2012 1 / 19 Outline

More information

Chapter 11 Map-Reduce, Hadoop, HDFS, Hbase, MongoDB, Apache HIVE, and Related

Chapter 11 Map-Reduce, Hadoop, HDFS, Hbase, MongoDB, Apache HIVE, and Related Chapter 11 Map-Reduce, Hadoop, HDFS, Hbase, MongoDB, Apache HIVE, and Related Summary Xiangzhe Li Nowadays, there are more and more data everyday about everything. For instance, here are some of the astonishing

More information

Big Data With Hadoop

Big Data With Hadoop With Saurabh Singh singh.903@osu.edu The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials

More information

Lucene in Action OTIS GOSPODNETIC ERIK HATCHER MANNING. Greenwich (74 w. long.)

Lucene in Action OTIS GOSPODNETIC ERIK HATCHER MANNING. Greenwich (74 w. long.) 2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. Lucene in Action OTIS GOSPODNETIC ERIK HATCHER i\ 11 MANNING Greenwich

More information

B+ Tree Properties B+ Tree Searching B+ Tree Insertion B+ Tree Deletion Static Hashing Extendable Hashing Questions in pass papers

B+ Tree Properties B+ Tree Searching B+ Tree Insertion B+ Tree Deletion Static Hashing Extendable Hashing Questions in pass papers B+ Tree and Hashing B+ Tree Properties B+ Tree Searching B+ Tree Insertion B+ Tree Deletion Static Hashing Extendable Hashing Questions in pass papers B+ Tree Properties Balanced Tree Same height for paths

More information

Efficiency of Web Based SAX XML Distributed Processing

Efficiency of Web Based SAX XML Distributed Processing Efficiency of Web Based SAX XML Distributed Processing R. Eggen Computer and Information Sciences Department University of North Florida Jacksonville, FL, USA A. Basic Computer and Information Sciences

More information

Analysis of Web Archives. Vinay Goel Senior Data Engineer

Analysis of Web Archives. Vinay Goel Senior Data Engineer Analysis of Web Archives Vinay Goel Senior Data Engineer Internet Archive Established in 1996 501(c)(3) non profit organization 20+ PB (compressed) of publicly accessible archival material Technology partner

More information

NoSQL Roadshow Berlin Kai Spichale

NoSQL Roadshow Berlin Kai Spichale Full-text Search with NoSQL Technologies NoSQL Roadshow Berlin Kai Spichale 25.04.2013 About me Kai Spichale Software Engineer at adesso AG Author in professional journals, conference speaker adesso is

More information

Big Data and Scripting. Part 4: Memory Hierarchies

Big Data and Scripting. Part 4: Memory Hierarchies 1, Big Data and Scripting Part 4: Memory Hierarchies 2, Model and Definitions memory size: M machine words total storage (on disk) of N elements (N is very large) disk size unlimited (for our considerations)

More information

Search and Real-Time Analytics on Big Data

Search and Real-Time Analytics on Big Data Search and Real-Time Analytics on Big Data Sewook Wee, Ryan Tabora, Jason Rutherglen Accenture & Think Big Analytics Strata New York October, 2012 Big Data: data becomes your core asset. It realizes its

More information

CSE-E5430 Scalable Cloud Computing Lecture 2

CSE-E5430 Scalable Cloud Computing Lecture 2 CSE-E5430 Scalable Cloud Computing Lecture 2 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 14.9-2015 1/36 Google MapReduce A scalable batch processing

More information

In Memory Accelerator for MongoDB

In Memory Accelerator for MongoDB In Memory Accelerator for MongoDB Yakov Zhdanov, Director R&D GridGain Systems GridGain: In Memory Computing Leader 5 years in production 100s of customers & users Starts every 10 secs worldwide Over 15,000,000

More information

Scalable Computing with Hadoop

Scalable Computing with Hadoop Scalable Computing with Hadoop Doug Cutting cutting@apache.org dcutting@yahoo-inc.com 5/4/06 Seek versus Transfer B-Tree requires seek per access unless to recent, cached page so can buffer & pre-sort

More information

A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM

A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM Sneha D.Borkar 1, Prof.Chaitali S.Surtakar 2 Student of B.E., Information Technology, J.D.I.E.T, sborkar95@gmail.com Assistant Professor, Information

More information

Apache HBase. Crazy dances on the elephant back

Apache HBase. Crazy dances on the elephant back Apache HBase Crazy dances on the elephant back Roman Nikitchenko, 16.10.2014 YARN 2 FIRST EVER DATA OS 10.000 nodes computer Recent technology changes are focused on higher scale. Better resource usage

More information

Integrating VoltDB with Hadoop

Integrating VoltDB with Hadoop The NewSQL database you ll never outgrow Integrating with Hadoop Hadoop is an open source framework for managing and manipulating massive volumes of data. is an database for handling high velocity data.

More information

BASHO DATA PLATFORM SIMPLIFIES BIG DATA, IOT, AND HYBRID CLOUD APPS

BASHO DATA PLATFORM SIMPLIFIES BIG DATA, IOT, AND HYBRID CLOUD APPS WHITEPAPER BASHO DATA PLATFORM BASHO DATA PLATFORM SIMPLIFIES BIG DATA, IOT, AND HYBRID CLOUD APPS INTRODUCTION Big Data applications and the Internet of Things (IoT) are changing and often improving our

More information

JReport Server Deployment Scenarios

JReport Server Deployment Scenarios JReport Server Deployment Scenarios Contents Introduction... 3 JReport Architecture... 4 JReport Server Integrated with a Web Application... 5 Scenario 1: Single Java EE Server with a Single Instance of

More information

Investigating Hadoop for Large Spatiotemporal Processing Tasks

Investigating Hadoop for Large Spatiotemporal Processing Tasks Investigating Hadoop for Large Spatiotemporal Processing Tasks David Strohschein dstrohschein@cga.harvard.edu Stephen Mcdonald stephenmcdonald@cga.harvard.edu Benjamin Lewis blewis@cga.harvard.edu Weihe

More information

Apache Hadoop. Alexandru Costan

Apache Hadoop. Alexandru Costan 1 Apache Hadoop Alexandru Costan Big Data Landscape No one-size-fits-all solution: SQL, NoSQL, MapReduce, No standard, except Hadoop 2 Outline What is Hadoop? Who uses it? Architecture HDFS MapReduce Open

More information

Hadoop Ecosystem B Y R A H I M A.

Hadoop Ecosystem B Y R A H I M A. Hadoop Ecosystem B Y R A H I M A. History of Hadoop Hadoop was created by Doug Cutting, the creator of Apache Lucene, the widely used text search library. Hadoop has its origins in Apache Nutch, an open

More information

Previous Lectures. B-Trees. External storage. Two types of memory. B-trees. Main principles

Previous Lectures. B-Trees. External storage. Two types of memory. B-trees. Main principles B-Trees Algorithms and data structures for external memory as opposed to the main memory B-Trees Previous Lectures Height balanced binary search trees: AVL trees, red-black trees. Multiway search trees:

More information

Information Retrieval Elasticsearch

Information Retrieval Elasticsearch Information Retrieval Elasticsearch IR Information retrieval (IR) is the activity of obtaining information resources relevant to an information need from a collection of information resources. Searches

More information

A programming model in Cloud: MapReduce

A programming model in Cloud: MapReduce A programming model in Cloud: MapReduce Programming model and implementation developed by Google for processing large data sets Users specify a map function to generate a set of intermediate key/value

More information

CSE 326: Data Structures B-Trees and B+ Trees

CSE 326: Data Structures B-Trees and B+ Trees Announcements (4//08) CSE 26: Data Structures B-Trees and B+ Trees Brian Curless Spring 2008 Midterm on Friday Special office hour: 4:-5: Thursday in Jaech Gallery (6 th floor of CSE building) This is

More information

A Performance Analysis of Distributed Indexing using Terrier

A Performance Analysis of Distributed Indexing using Terrier A Performance Analysis of Distributed Indexing using Terrier Amaury Couste Jakub Kozłowski William Martin Indexing Indexing Used by search

More information

Full Text Search in MySQL 5.1 New Features and HowTo

Full Text Search in MySQL 5.1 New Features and HowTo Full Text Search in MySQL 5.1 New Features and HowTo Alexander Rubin Senior Consultant, MySQL AB 1 Full Text search Natural and popular way to search for information Easy to use: enter key words and get

More information

Mobile Storage and Search Engine of Information Oriented to Food Cloud

Mobile Storage and Search Engine of Information Oriented to Food Cloud Advance Journal of Food Science and Technology 5(10): 1331-1336, 2013 ISSN: 2042-4868; e-issn: 2042-4876 Maxwell Scientific Organization, 2013 Submitted: May 29, 2013 Accepted: July 04, 2013 Published:

More information

PUBMED: an efficient biomedical based hierarchical search engine ABSTRACT:

PUBMED: an efficient biomedical based hierarchical search engine ABSTRACT: PUBMED: an efficient biomedical based hierarchical search engine ABSTRACT: Search queries on biomedical databases, such as PubMed, often return a large number of results, only a small subset of which is

More information

CatDV Pro Workgroup Serve r

CatDV Pro Workgroup Serve r Architectural Overview CatDV Pro Workgroup Server Square Box Systems Ltd May 2003 The CatDV Pro client application is a standalone desktop application, providing video logging and media cataloging capability

More information

Katta & Hadoop. Katta - Distributed Lucene Index in Production. Stefan Groschupf Scale Unlimited, 101tec. sg{at}101tec.com

Katta & Hadoop. Katta - Distributed Lucene Index in Production. Stefan Groschupf Scale Unlimited, 101tec. sg{at}101tec.com 1 Katta & Hadoop Katta - Distributed Lucene Index in Production Stefan Groschupf Scale Unlimited, 101tec. sg{at}101tec.com foto by: belgianchocolate@flickr.com 2 Intro Business intelligence reports from

More information

B-Trees. Algorithms and data structures for external memory as opposed to the main memory B-Trees. B -trees

B-Trees. Algorithms and data structures for external memory as opposed to the main memory B-Trees. B -trees B-Trees Algorithms and data structures for external memory as opposed to the main memory B-Trees Previous Lectures Height balanced binary search trees: AVL trees, red-black trees. Multiway search trees:

More information

Data processing goes big

Data processing goes big Test report: Integration Big Data Edition Data processing goes big Dr. Götz Güttich Integration is a powerful set of tools to access, transform, move and synchronize data. With more than 450 connectors,

More information

Large Scale Text Analysis Using the Map/Reduce

Large Scale Text Analysis Using the Map/Reduce Large Scale Text Analysis Using the Map/Reduce Hierarchy David Buttler This work is performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract

More information

InfiniteGraph: The Distributed Graph Database

InfiniteGraph: The Distributed Graph Database A Performance and Distributed Performance Benchmark of InfiniteGraph and a Leading Open Source Graph Database Using Synthetic Data Objectivity, Inc. 640 West California Ave. Suite 240 Sunnyvale, CA 94086

More information

Componize DITA CMS 1.6 List of features

Componize DITA CMS 1.6 List of features Componize DITA CMS 1.6 List of features Componize provides a solution to facilitate authoring and controlling large volumes of structured modular content. Authoring... 1 Componize Author Page... 2 Metadata

More information

DataStax Enterprise 3.x

DataStax Enterprise 3.x DataStax Enterprise 3.x Realtime Analytics with Solr Jason Rutherglen 2012 DataStax 1 About the Presenter Big Data Engineer at DataStax Co-author of Programming Hive and Lucene and Solr: The Definitive

More information

Search & Export Report Data

Search & Export Report Data Search & Export Report Data Version: Draft 2 Abstract This document describes how to export the data from a saved report document. Document Revisions Version Date Description of Changes Draft 2 08/12/2005

More information

Hypertable Architecture Overview

Hypertable Architecture Overview WHITE PAPER - MARCH 2012 Hypertable Architecture Overview Hypertable is an open source, scalable NoSQL database modeled after Bigtable, Google s proprietary scalable database. It is written in C++ for

More information

SharePoint Server 2010 Capacity Management: Software Boundaries and Limits

SharePoint Server 2010 Capacity Management: Software Boundaries and Limits SharePoint Server 2010 Capacity Management: Software Boundaries and s This document is provided as-is. Information and views expressed in this document, including URL and other Internet Web site references,

More information

HDFS. Hadoop Distributed File System

HDFS. Hadoop Distributed File System HDFS Kevin Swingler Hadoop Distributed File System File system designed to store VERY large files Streaming data access Running across clusters of commodity hardware Resilient to node failure 1 Large files

More information

Accelerating Hadoop MapReduce Using an In-Memory Data Grid

Accelerating Hadoop MapReduce Using an In-Memory Data Grid Accelerating Hadoop MapReduce Using an In-Memory Data Grid By David L. Brinker and William L. Bain, ScaleOut Software, Inc. 2013 ScaleOut Software, Inc. 12/27/2012 H adoop has been widely embraced for

More information

CSE454 Project Part4: Dealer s Choice Assigned: Monday, November 28, 2005 Due: 10:30 AM, Thursday, December 15, 2005

CSE454 Project Part4: Dealer s Choice Assigned: Monday, November 28, 2005 Due: 10:30 AM, Thursday, December 15, 2005 CSE454 Project Part4: Dealer s Choice Assigned: Monday, November 28, 2005 Due: 10:30 AM, Thursday, December 15, 2005 1 Project Description For the last part of your project, you should choose what to do.

More information

Apache HBase: the Hadoop Database

Apache HBase: the Hadoop Database Apache HBase: the Hadoop Database Yuanru Qian, Andrew Sharp, Jiuling Wang Today we will discuss Apache HBase, the Hadoop Database. HBase is designed specifically for use by Hadoop, and we will define Hadoop

More information

EFFECTIVE STRATEGIES FOR SEARCHING ORACLE UCM. Alan Mackenthun Senior Software Consultant 4/23/2010. F i s h b o w l S o l u t I o n s

EFFECTIVE STRATEGIES FOR SEARCHING ORACLE UCM. Alan Mackenthun Senior Software Consultant 4/23/2010. F i s h b o w l S o l u t I o n s EFFECTIVE STRATEGIES FOR SEARCHING ORACLE UCM Alan Mackenthun Senior Software Consultant 4/23/2010 F i s h b o w l S o l u t I o n s EFFECTIVE STRATEGIES FOR SEARCHING ORACLE UCM Contents INTRODUCTION...

More information

Introduction to Hadoop HDFS and Ecosystems. Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data

Introduction to Hadoop HDFS and Ecosystems. Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data Introduction to Hadoop HDFS and Ecosystems ANSHUL MITTAL Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data Topics The goal of this presentation is to give

More information

Big Data and Apache Hadoop s MapReduce

Big Data and Apache Hadoop s MapReduce Big Data and Apache Hadoop s MapReduce Michael Hahsler Computer Science and Engineering Southern Methodist University January 23, 2012 Michael Hahsler (SMU/CSE) Hadoop/MapReduce January 23, 2012 1 / 23

More information

So today we shall continue our discussion on the search engines and web crawlers. (Refer Slide Time: 01:02)

So today we shall continue our discussion on the search engines and web crawlers. (Refer Slide Time: 01:02) Internet Technology Prof. Indranil Sengupta Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture No #39 Search Engines and Web Crawler :: Part 2 So today we

More information

Using Apache Solr for Ecommerce Search Applications

Using Apache Solr for Ecommerce Search Applications Using Apache Solr for Ecommerce Search Applications Rajani Maski Happiest Minds, IT Services SHARING. MINDFUL. INTEGRITY. LEARNING. EXCELLENCE. SOCIAL RESPONSIBILITY. 2 Copyright Information This document

More information

A. Aiken & K. Olukotun PA3

A. Aiken & K. Olukotun PA3 Programming Assignment #3 Hadoop N-Gram Due Tue, Feb 18, 11:59PM In this programming assignment you will use Hadoop s implementation of MapReduce to search Wikipedia. This is not a course in search, so

More information

The Open Source Knowledge Discovery and Document Analysis Platform

The Open Source Knowledge Discovery and Document Analysis Platform Enabling Agile Intelligence through Open Analytics The Open Source Knowledge Discovery and Document Analysis Platform 17/10/2012 1 Agenda Introduction and Agenda Problem Definition Knowledge Discovery

More information

Hadoop MapReduce over Lustre* High Performance Data Division Omkar Kulkarni April 16, 2013

Hadoop MapReduce over Lustre* High Performance Data Division Omkar Kulkarni April 16, 2013 Hadoop MapReduce over Lustre* High Performance Data Division Omkar Kulkarni April 16, 2013 * Other names and brands may be claimed as the property of others. Agenda Hadoop Intro Why run Hadoop on Lustre?

More information

Performance rule violations usually result in increased CPU or I/O, time to fix the mistake, and ultimately, a cost to the business unit.

Performance rule violations usually result in increased CPU or I/O, time to fix the mistake, and ultimately, a cost to the business unit. Is your database application experiencing poor response time, scalability problems, and too many deadlocks or poor application performance? One or a combination of zparms, database design and application

More information

Jeffrey D. Ullman slides. MapReduce for data intensive computing

Jeffrey D. Ullman slides. MapReduce for data intensive computing Jeffrey D. Ullman slides MapReduce for data intensive computing Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very

More information

Facebook: Cassandra. Smruti R. Sarangi. Department of Computer Science Indian Institute of Technology New Delhi, India. Overview Design Evaluation

Facebook: Cassandra. Smruti R. Sarangi. Department of Computer Science Indian Institute of Technology New Delhi, India. Overview Design Evaluation Facebook: Cassandra Smruti R. Sarangi Department of Computer Science Indian Institute of Technology New Delhi, India Smruti R. Sarangi Leader Election 1/24 Outline 1 2 3 Smruti R. Sarangi Leader Election

More information

Introduction to Parallel Programming and MapReduce

Introduction to Parallel Programming and MapReduce Introduction to Parallel Programming and MapReduce Audience and Pre-Requisites This tutorial covers the basics of parallel programming and the MapReduce programming model. The pre-requisites are significant

More information

Hadoop and Map-Reduce. Swati Gore

Hadoop and Map-Reduce. Swati Gore Hadoop and Map-Reduce Swati Gore Contents Why Hadoop? Hadoop Overview Hadoop Architecture Working Description Fault Tolerance Limitations Why Map-Reduce not MPI Distributed sort Why Hadoop? Existing Data

More information

Information Retrieval Systems in XML Based Database A review

Information Retrieval Systems in XML Based Database A review Information Retrieval Systems in XML Based Database A review Preeti Pandey 1, L.S.Maurya 2 Research Scholar, IT Department, SRMSCET, Bareilly, India 1 Associate Professor, IT Department, SRMSCET, Bareilly,

More information

Big Systems, Big Data

Big Systems, Big Data Big Systems, Big Data When considering Big Distributed Systems, it can be noted that a major concern is dealing with data, and in particular, Big Data Have general data issues (such as latency, availability,

More information

Spark ΕΡΓΑΣΤΗΡΙΟ 10. Prepared by George Nikolaides 4/19/2015 1

Spark ΕΡΓΑΣΤΗΡΙΟ 10. Prepared by George Nikolaides 4/19/2015 1 Spark ΕΡΓΑΣΤΗΡΙΟ 10 Prepared by George Nikolaides 4/19/2015 1 Introduction to Apache Spark Another cluster computing framework Developed in the AMPLab at UC Berkeley Started in 2009 Open-sourced in 2010

More information

A Novel Cloud Based Elastic Framework for Big Data Preprocessing

A Novel Cloud Based Elastic Framework for Big Data Preprocessing School of Systems Engineering A Novel Cloud Based Elastic Framework for Big Data Preprocessing Omer Dawelbeit and Rachel McCrindle October 21, 2014 University of Reading 2008 www.reading.ac.uk Overview

More information

Inmagic Content Server Workgroup Configuration Technical Guidelines

Inmagic Content Server Workgroup Configuration Technical Guidelines Inmagic Content Server Workgroup Configuration Technical Guidelines 6/2005 Page 1 of 12 Inmagic Content Server Workgroup Configuration Technical Guidelines Last Updated: June, 2005 Inmagic, Inc. All rights

More information

Using EMC Documentum with Adobe LiveCycle ES

Using EMC Documentum with Adobe LiveCycle ES Technical Guide Using EMC Documentum with Adobe LiveCycle ES Table of contents 1 Deployment 3 Managing LiveCycle ES development assets in Documentum 5 Developing LiveCycle applications with contents in

More information

Take An Internal Look at Hadoop. Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com

Take An Internal Look at Hadoop. Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com What s Hadoop Framework for running applications on large clusters of commodity hardware Scale: petabytes of data

More information

The Hadoop Framework

The Hadoop Framework The Hadoop Framework Nils Braden University of Applied Sciences Gießen-Friedberg Wiesenstraße 14 35390 Gießen nils.braden@mni.fh-giessen.de Abstract. The Hadoop Framework offers an approach to large-scale

More information

Deploying Web Applications with Eclipse and Tomcat

Deploying Web Applications with Eclipse and Tomcat Deploying Web Applications with Eclipse and Tomcat coreservlets.com custom onsite training For customized training related to JavaScript or Java, email hall@coreservlets.com Marty is also available for

More information

Distributed Computing" with Open-Source Software

Distributed Computing with Open-Source Software Distributed Computing" with Open-Source Software Reza Zadeh Presented at Infosys OSSmosis Problem Data growing faster than processing speeds Only solution is to parallelize on large clusters» Wide use

More information

Couchbase Server Under the Hood

Couchbase Server Under the Hood Couchbase Server Under the Hood An Architectural Overview Couchbase Server is an open-source distributed NoSQL document-oriented database for interactive applications, uniquely suited for those needing

More information

Package Designer Guide

Package Designer Guide Package Designer Guide Rev: 13 December 2011 Sitecore CMS 6.5 Package Designer Guide An administrator's guide to creating and editing Sitecore packages Table of Contents Chapter 1 Introduction... 3 Chapter

More information

Content Management Implementation Guide 5.3 SP1

Content Management Implementation Guide 5.3 SP1 SDL Tridion R5 Content Management Implementation Guide 5.3 SP1 Read this document to implement and learn about the following Content Manager features: Publications Blueprint Publication structure Users

More information

Search Big Data with MySQL and Sphinx. Mindaugas Žukas www.ivinco.com

Search Big Data with MySQL and Sphinx. Mindaugas Žukas www.ivinco.com Search Big Data with MySQL and Sphinx Mindaugas Žukas www.ivinco.com Agenda Big Data Architecture Factors and Technologies MySQL and Big Data Sphinx Search Server overview Case study: building a Big Data

More information

Scalable Forensics with TSK and Hadoop. Jon Stewart

Scalable Forensics with TSK and Hadoop. Jon Stewart Scalable Forensics with TSK and Hadoop Jon Stewart CPU Clock Speed Hard Drive Capacity The Problem CPU clock speed stopped doubling Hard drive capacity kept doubling Multicore CPUs to the rescue!...but

More information

WIRIS quizzes web services Getting started with PHP and Java

WIRIS quizzes web services Getting started with PHP and Java WIRIS quizzes web services Getting started with PHP and Java Document Release: 1.3 2011 march, Maths for More www.wiris.com Summary This document provides client examples for PHP and Java. Contents WIRIS

More information

How to Choose Between Hadoop, NoSQL and RDBMS

How to Choose Between Hadoop, NoSQL and RDBMS How to Choose Between Hadoop, NoSQL and RDBMS Keywords: Jean-Pierre Dijcks Oracle Redwood City, CA, USA Big Data, Hadoop, NoSQL Database, Relational Database, SQL, Security, Performance Introduction A

More information

Associate Professor, Department of CSE, Shri Vishnu Engineering College for Women, Andhra Pradesh, India 2

Associate Professor, Department of CSE, Shri Vishnu Engineering College for Women, Andhra Pradesh, India 2 Volume 6, Issue 3, March 2016 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Special Issue

More information

Hadoop Streaming. Table of contents

Hadoop Streaming. Table of contents Table of contents 1 Hadoop Streaming...3 2 How Streaming Works... 3 3 Streaming Command Options...4 3.1 Specifying a Java Class as the Mapper/Reducer... 5 3.2 Packaging Files With Job Submissions... 5

More information

Glassfish, JAVA EE, Servlets, JSP, EJB

Glassfish, JAVA EE, Servlets, JSP, EJB Glassfish, JAVA EE, Servlets, JSP, EJB Java platform A Java platform comprises the JVM together with supporting class libraries. Java 2 Standard Edition (J2SE) (1999) provides core libraries for data structures,

More information

Hadoop Distributed Filesystem. Spring 2015, X. Zhang Fordham Univ.

Hadoop Distributed Filesystem. Spring 2015, X. Zhang Fordham Univ. Hadoop Distributed Filesystem Spring 2015, X. Zhang Fordham Univ. MapReduce Programming Model Split Shuffle Input: a set of [key,value] pairs intermediate [key,value] pairs [k1,v11,v12, ] [k2,v21,v22,

More information

Hadoop Distributed File System. Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com

Hadoop Distributed File System. Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com Hadoop Distributed File System Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com Hadoop, Why? Need to process huge datasets on large clusters of computers

More information

Finding the Needle in a Big Data Haystack. Wolfgang Hoschek (@whoschek) JAX 2014

Finding the Needle in a Big Data Haystack. Wolfgang Hoschek (@whoschek) JAX 2014 Finding the Needle in a Big Data Haystack Wolfgang Hoschek (@whoschek) JAX 2014 1 About Wolfgang Software Engineer @ Cloudera Search Platform Team Previously CERN, Lawrence Berkeley National Laboratory,

More information

HADOOP MOCK TEST HADOOP MOCK TEST I

HADOOP MOCK TEST HADOOP MOCK TEST I http://www.tutorialspoint.com HADOOP MOCK TEST Copyright tutorialspoint.com This section presents you various set of Mock Tests related to Hadoop Framework. You can download these sample mock tests at

More information

Making Big Data Processing Simple with Spark. Matei Zaharia

Making Big Data Processing Simple with Spark. Matei Zaharia Making Big Data Processing Simple with Spark Matei Zaharia December 17, 2015 What is Apache Spark? Fast and general cluster computing engine that generalizes the MapReduce model Makes it easy and fast

More information

Hadoop-based Open Source ediscovery: FreeEed. (Easy as popcorn)

Hadoop-based Open Source ediscovery: FreeEed. (Easy as popcorn) + Hadoop-based Open Source ediscovery: FreeEed (Easy as popcorn) + Hello! 2 Sujee Maniyam & Mark Kerzner Founders @ Elephant Scale consulting and training around Hadoop, Big Data technologies Enterprise

More information

Research on Clustering Analysis of Big Data Yuan Yuanming 1, 2, a, Wu Chanle 1, 2

Research on Clustering Analysis of Big Data Yuan Yuanming 1, 2, a, Wu Chanle 1, 2 Advanced Engineering Forum Vols. 6-7 (2012) pp 82-87 Online: 2012-09-26 (2012) Trans Tech Publications, Switzerland doi:10.4028/www.scientific.net/aef.6-7.82 Research on Clustering Analysis of Big Data

More information

Search and Information Retrieval

Search and Information Retrieval Search and Information Retrieval Search on the Web 1 is a daily activity for many people throughout the world Search and communication are most popular uses of the computer Applications involving search

More information

CiteSeer x in the Cloud

CiteSeer x in the Cloud Published in the 2nd USENIX Workshop on Hot Topics in Cloud Computing 2010 CiteSeer x in the Cloud Pradeep B. Teregowda Pennsylvania State University C. Lee Giles Pennsylvania State University Bhuvan Urgaonkar

More information

Physical Data Organization

Physical Data Organization Physical Data Organization Database design using logical model of the database - appropriate level for users to focus on - user independence from implementation details Performance - other major factor

More information

Sector vs. Hadoop. A Brief Comparison Between the Two Systems

Sector vs. Hadoop. A Brief Comparison Between the Two Systems Sector vs. Hadoop A Brief Comparison Between the Two Systems Background Sector is a relatively new system that is broadly comparable to Hadoop, and people want to know what are the differences. Is Sector

More information

Increasing Database Performance using Indexes

Increasing Database Performance using Indexes Database Systems Journal vol. II, no. 2/2011 13 Cecilia CIOLOCA, Mihai GEORGESCU Economic Informatics Department, Academy of Economic Studies Bucharest, ROMANIA cecilia_cioloca@yahoo.com, mihai.georgescu@europe.com

More information

Xtreeme Search Engine Studio Help. 2007 Xtreeme

Xtreeme Search Engine Studio Help. 2007 Xtreeme Xtreeme Search Engine Studio Help 2007 Xtreeme I Search Engine Studio Help Table of Contents Part I Introduction 2 Part II Requirements 4 Part III Features 7 Part IV Quick Start Tutorials 9 1 Steps to

More information

Electronic Document Management Using Inverted Files System

Electronic Document Management Using Inverted Files System EPJ Web of Conferences 68, 0 00 04 (2014) DOI: 10.1051/ epjconf/ 20146800004 C Owned by the authors, published by EDP Sciences, 2014 Electronic Document Management Using Inverted Files System Derwin Suhartono,

More information

CSE 326, Data Structures. Sample Final Exam. Problem Max Points Score 1 14 (2x7) 2 18 (3x6) 3 4 4 7 5 9 6 16 7 8 8 4 9 8 10 4 Total 92.

CSE 326, Data Structures. Sample Final Exam. Problem Max Points Score 1 14 (2x7) 2 18 (3x6) 3 4 4 7 5 9 6 16 7 8 8 4 9 8 10 4 Total 92. Name: Email ID: CSE 326, Data Structures Section: Sample Final Exam Instructions: The exam is closed book, closed notes. Unless otherwise stated, N denotes the number of elements in the data structure

More information

Operating Systems: Internals and Design Principles. Chapter 12 File Management Seventh Edition By William Stallings

Operating Systems: Internals and Design Principles. Chapter 12 File Management Seventh Edition By William Stallings Operating Systems: Internals and Design Principles Chapter 12 File Management Seventh Edition By William Stallings Operating Systems: Internals and Design Principles If there is one singular characteristic

More information

Spring,2015. Apache Hive BY NATIA MAMAIASHVILI, LASHA AMASHUKELI & ALEKO CHAKHVASHVILI SUPERVAIZOR: PROF. NODAR MOMTSELIDZE

Spring,2015. Apache Hive BY NATIA MAMAIASHVILI, LASHA AMASHUKELI & ALEKO CHAKHVASHVILI SUPERVAIZOR: PROF. NODAR MOMTSELIDZE Spring,2015 Apache Hive BY NATIA MAMAIASHVILI, LASHA AMASHUKELI & ALEKO CHAKHVASHVILI SUPERVAIZOR: PROF. NODAR MOMTSELIDZE Contents: Briefly About Big Data Management What is hive? Hive Architecture Working

More information

Web Search Engines. Search Engine Characteristics. Web Search Queries. Chapter 27, Part C Based on Larson and Hearst s slides at UC-Berkeley

Web Search Engines. Search Engine Characteristics. Web Search Queries. Chapter 27, Part C Based on Larson and Hearst s slides at UC-Berkeley Web Search Engines Chapter 27, Part C Based on Larson and Hearst s slides at UC-Berkeley http://www.sims.berkeley.edu/courses/is202/f00/ Database Management Systems, R. Ramakrishnan 1 Search Engine Characteristics

More information

Preparing a SQL Server for EmpowerID installation

Preparing a SQL Server for EmpowerID installation Preparing a SQL Server for EmpowerID installation By: Jamis Eichenauer Last Updated: October 7, 2014 Contents Hardware preparation... 3 Software preparation... 3 SQL Server preparation... 4 Full-Text Search

More information

Developing a MapReduce Application

Developing a MapReduce Application TIE 12206 - Apache Hadoop Tampere University of Technology, Finland November, 2014 Outline 1 MapReduce Paradigm 2 Hadoop Default Ports 3 Outline 1 MapReduce Paradigm 2 Hadoop Default Ports 3 MapReduce

More information