Image Management Suite. Mini Thesis. Roland Foster. Supervisors: Mr. Mehrdad Ghaziasgar and Mr. James Connan. B.Sc. Honours

Transcription

1 Image Management Suite Mini Thesis Roland Foster Supervisors: Mr. Mehrdad Ghaziasgar and Mr. James Connan B.Sc. Honours Department of Computer Science 2012

2 Acknowledgements I would like to thank my parents and siblings for all their support and motivation throughout the years of my studies. I also thank my parents for supporting me financially during my studies at the University of the Western Cape which allowed me to fully focus on my studies. A special thanks goes to my supervisors Mr. M. Ghaziasgar and Mr. J. Connan, for their support and sound advice and also for having the confidence in me to complete this project. I also express my gratitude to Mr. W. Nel for his help and mentorship throughout the progression of this year.

3 Abstract Photos store the precious memories we experience throughout our lives. They hold the memories of different moments in our lives from birthdays, to weddings, to graduation ceremonies etc. We first stored all of our photographs in photo albums but now these photographs are now being stored on our computers thanks to digital images. The storing of our photographs on the computer makes it difficult for us to remember where specific photographs are located. The aim of the project is to create an Image Management Suite. It is an application which allows the easy organising and retrieval of digital images and their metadata.

4 CONTENTS PAGE CHAPTER 1: INTRODUCTION 6 CHAPTER 2: USER REQUIREMENTS User s view of the problem Duplicate Images Searching for images Tags for images Tagging detected objects Hardware and software requirements for the proposed application Description of the problem Duplicate images Searching for images Tags for images Expectations from the software solution 9 CHAPTER 3: REQUIREMENTS ANALYSIS The Designer s Interpretation of the Problem Breakdown and Analysis of the Problem Remove Duplicate Images Object Detection Adding Tags Current Solutions Suggested Solution 14 CHAPTER 4: USER INTERFACE SPECIFICATION Introduction Description of the user interface Breakdown of the user interface Menu bar 16 CHAPTER 5: HIGH LEVEL DESIGN Introduction Searching for images on the hard drive Tagging an image Searching for images based on tags Automatic Tags Detection Data Dictionary 20

5 CHAPTER 6: LOW LEVEL DESIGN Searching for images and eliminating duplicates Manual Tagging Searching for images based on tags Automatic Tags Detection 25 CHAPTER 8: USER MANUAL Initial Start-up Viewing an image Tagging an image Co-ordinate Based tagging Used Tags Tagging Non-co-ordinate Based Tagging Searching for an image based on tags Viewing Position Tags Automatic Tagging Importing a haarcascade Editing tags Deleting tags 42 CHAPTER 9: CODE DOCUMENTATION 43 CONCLUSION 44 REFERENCES 45

6 Chapter 1: Introduction The aim of this project is to develop an image management suite which mitigates duplicate images and will make it easier for a user retrieve a desired image. The application will make use of embedded tags to their true potential by improving the manner in which these tags are used when searching for an image. It will also provide the user with better tagging options and an improved approach to classifying an image. With the aid of image processing techniques of the OpenCV we can use haarcascades to automatically detect objects in an image. By using haarcascades one can detect objects such as faces within an image and by making use of embedded tags we can automatically detect and tag these objects.

7 Chapter 2: User Requirements This chapter discusses the problem from the user s point of view. It describes the functionality which the user requires from the proposed software solution. 2.2 User s view of the problem Duplicate Images When having a large collection of images on a computer, there is a possibility of having many duplicate images in this collection. These duplicates could occur when the user transfers images from a camera or a storage device to his/her computer. Some of the images which are being transferred are images which the user already has on his/her computer. For this reason users often end up having multiple duplicates of images on their computers Searching for images Searching for a specific image from a computer which contains thousands of images is a difficult task. If the user does not remember the name or path of the image the user would have to navigate the computer s entire filing system, opening all the folders and each image individually to find the desired image. The user requires the application to search for image based on search criteria from the user Tags for images In the past users wrote comments on the back of their printed photographs to describe the contents of that specific photograph. The user would write the names of the people in the image on the back of the card. This functionality is to be added to the images themselves instead of adding it to a separate database to ensure portability

8 2.2.4 Tagging detected objects A professional photographer takes thousands of photos in a month and does not have the time to tag every single image. The user requires an application which can detect basic objects in an image and automatically add these tags to the image. He/she would now be provided with some basic tags making it easier for the user to further classify the image Hardware and software requirements for the proposed application A computer with Windows 7 or Linux based operating systems Open CV 2.2 (Open Source Computer Vision Library) Python Pyexiv2 - A python module that allows python scripts to read and write metadata (EXIF, IPTC, XMP, thumbnails) embedded in image files. [3] wxpython - A python wrapper for the WxWidgets GUI toolkit 2.3 Description of the problem Duplicate images When transferring images from a camera to a computer the user usually copies the folder from the camera s storage device which contains the images. The next time the user takes photos, he/she does not delete the previous photos on the camera. The images are then copied from the camera to a new folder on the computer. By doing this the user is copying duplicate images to his/her computer. If the user continues doing this he/she will have multiple duplicates of images on his/her computer. It is a tedious task for the user to manually find all duplicate images and delete them.

9 2.3.2 Searching for images The user usually finds an image by remembering where the image was stored and navigating to the folder in which the image is stored. If the user cannot remember the path to the desired image, he/she can search for the image based on its file name using the search utility provided by the operating system of his/her computer. The problem occurs when a user is searching for a particular image and he/she cannot remember the path to the image file or the name of the file. The user only remembers what the contents of the image are Tags for images Before users stored images digitally they would write on the back of the photo to describe the contents of the image. This functionality is to be added to the images themselves instead of adding it to a separate database to ensure portability Tagging detected objects When having a large collection of images it is a tedious task to label each image with tags. In order to make life easier for the user one must automatically detect objects in an image. 2.4 Expectations from the software solution The proposed software solution will eliminate duplicate images, allow the user to add tags to images and easily retrieve images from a computer s hard drive based on a search criteria entered by the user. By using simple search terms such as name, place, occasion etc, the user will be able to search for images on his/her computer. The application will automatically detect the objects such as faces, cars and wall clocks and add them as tags to the image. The application s GUI (Graphical User Interface) should be user friendly and be compatible with Windows and Linux based operating systems.

10 2.4.1 Features of the software solution: Eliminate Duplicate Images A function will automatically run on the initial start up of the application and also when prompted by the user to search the user s hard drive for all images files. The function will find all image files on the computer and mitigate the problem of duplicate images. Adding Tags The user will be able to label the contents of the images with tags. Searching For An Image The user should be to search for an image not based on filename or path of the file but by entering key words. The application will then search for images which match these key words. For instance, if a person named John is searching for an image of himself at the V&A Waterfront during the evening; the application must find all images related to John at the V&A Waterfront during the evening. Object Detection The application will also implement object detection. For example, if there is a face in an image the application will automatically detect the face and add it to the tags of the image.

11 Chapter 3: Requirements Analysis This chapter takes the user requirements defined in chapter 2 and explains how the designer interprets them. It analyses the user requirements and breaks these requirements down into parts. 3.1 The Designer s Interpretation of the Problem If there are duplicate images on the computer of the user, he/she would not know which image amongst the duplicates he/she had added the tags to. To find the image containing the tags he/she would have to open each of the duplicate images individually. Image processing techniques provided from Open CV require a lot of computing power to detect objects in an image. If the duplicate images are eliminated it decreases the number of images and ultimately lowers the time required to detect objects for all images. The proposed solution will automatically detect basic objects in the image such as a person or car and add these as tags to the image. Since these images will now have tags it makes searching for them easier. The application receives search criteria for the user and searches for images which meet these criteria. It does this by searching through a combination of the detected tags which the application automatically added as well as the tags entered by the user Software and programming tools needed to develop the application Open CV (Open Source Computer Vision Library): A computer vision library which supports real-time image processing. Open CV will be used to implement object detection. [1] PyScripter: An open-source Python Integrated Development Environment (IDE). [2] Python:

12 An object orientated programming language. The application will be programmed using Python. wxpython: A python wrapper for the WxWidgets GUI toolkit. Wx Widgets is GUI (Graphical User Interface) toolkit for the Python programming language. Pyexiv2: A python module that allows python scripts to read and write metadata (EXIF, IPTC, XMP, thumbnails) embedded in image files. [3] 3.2 Breakdown and Analysis of the Problem Remove Duplicate Images To solve this problem the application will run a search function which finds all image files on the user s computer and creates a list of images without these duplicates. To find the duplicate images generate checksums for the images and compare the checksums of each image against one another to find the duplicates. The application does not delete these duplicates but it restricts the user to this list of images Object Detection The user has to manually enter tags in for each image and if there are thousands of images it will be a time consuming task to label all these images. By making use of the Open CV Library the application will implement object detection and use Haar-like features [1] to detect basic objects on each image from the list generated. For instance, if the there is a face in an image the application will automatically detect the face. Once these objects are detected they will be embedded within the image Adding Tags

13 The application will allow the user to add descriptive tags to the image such as name, place, occasion and comments. The descriptive tags entered by the user as well as the objects detected will be embedded within the image. Pyexiv2 [1] will be used to embed the tags within the images. 3.3 Current Solutions To my knowledge there are no applications which provide the above mentioned functionality as an all-in-one application. There are however, stand-alone applications which provide the above mentioned features but most of these are expensive, proprietary software and platform dependent. A few examples of current solutions: Windows Live Photo Gallery An application included in the Windows 7 operating system. It is a photo viewer which allows the user to add descriptive tags to images. Picasa A photo manager in which users can view, edit and add tags to images. itag A tool that embeds title, description and keywords into your photos and movies. Google Image Search Allows a user to search for images from the internet Remove Duplicate Pictures A duplicate picture remover automatically removes duplicate images.

14 3.4 Suggested Solution The solution will be a cost effective one since the programming software and tools used are all open source. The application will be able to run on the Windows and Linux operating systems.

15 Chapter 4: User Interface Specification 4.1 Introduction In this chapter the User Interface Specification of the Image Management Suite is discussed. A description of what the user interface will look like is given and the manner in which the user interacts with the application is explained. Screen shots of the proposed user interface are included and broken down into parts to discuss each element of the interface. 4.2 Description of the user interface The user interface for the Image Management Suite is a GUI (Graphical User Interface) as shown below: Breakdown of the user interface Static Image: The image is displayed here. By default there is a black background image when no image is selected. Search Button and Text Control: The user enters in keywords to search for an image based on its tags and clicks on the search button to execute the command ListBox: The tags of the image are displayed here. Add Tags Button: The user enters in text in the text box next to the button and clicks the button to add tags to an image. Figure 1: A screen shot of the user interface for the Image Management Suite. File Tree: A list of all the images on the user s hard drive are displayed in this tree.

16 4.2.2 Menu bar The menu bar consists of three menu items namely File, View and Help. These items will be discussed in the screen shots below Menu: File Menu item: Quit Click Quit for exiting the application Menu: View Menu item: Clear List Clear List clears the list of images from the list box so that the user can run the image search function again.

17 Chapter 5: High Level Design 5.1 Introduction The High Level Design is discussed in this chapter. It provides an object-orientated or abstract view of the proposed application, briefly explaining the objects and providing a data dictionary for each object. A class diagram for the Image Management Suite is shown Searching for images on the hard drive The computer is searched to find image files on its hard drive. The duplicate images are then removed.

18 5.1.2 Tagging an image Entered Tags XML string construction Embed XML within the image As shown in the above figure the user enters a tag, the tag is then constructed into an XML string and this string is then added into the image Searching for images based on tags Search terms Match the search terms against the tags extracted from the images Images matching search criteria are returned As shown above the user enters the search terms and these terms are then matched against the tags extracted from the images. The images which match the search terms entered are then displayed to the user.

19 5.1.4 Automatic Tags Detection Detect and label objects Face Wall Clock By making use of the OpenCV libraries we detect haar-like features in images. The haarcascades of a Face, Car and Wall Clock are included with the software solution. As shown in the image, the application automatically detects the features of a face using the face haarcascade and it then detects the features of a wall clock using the wall clock haarcascade.

20 5.3 Data Dictionary An explanation of each class is given in the data dictionary below: Object Description IMS (Image Management Suite) The IMS class builds the Graphical User Interface of the application and the widgets (e.g. buttons, menu bars and text controls). It ties together all the other definitions/methods of the system into one application. The user will interact with this class. readexifdata Has a definition/method called getexifdata() which fetches the EXIF tags data which is embedded within an image. It then parses the EXIF data into an XML tree and returns the tags to in a dictionary data structure. FetchChecksum Has the definition/method called getchecksum() which returns the checksum embedded within the image. tagsearch Has the definition/method called tagfinder which finds images containing tags relating to the search criteria entered by the user.

21 imgsearch A search definition/method which searches for all images files types and writes these images filenames to a text file. (Images formats include:.jpg,.tiff,.png). The text file is used to generate the list of images in the list box of the GUI. onfind Prompts the user to select the directory selection box. It then searches for all images, removes duplicates and builds a tree with all images sorted in subcategories: All- all images in given directory Face- All images containing a face Car- All images containing a car Wall Clock - All images containing a wall clock onautotagging Provides the user with a selection of objects to be detected in all images. The user clicks a checkbox to select for each object he/she would like to have detected in the images. It then runs the cv.haardetectobjects method which makes use of xml haarcascade files to detect the faces, cars and wall clocks in the images. The objects which are detected are then constructed into an

22 XML string with their co-ordinates and embedded within the image. detectobj importhaar2 Makes use of xml haarcascade files and cv.haardetectobjects to detect the faces, cars and wall clocks in the images. The detected objects and their coordinates are then formed into an XML string with and embedded within the image. A dialog box prompting the user to set the file path of the haarcascade XML file, the name of the object to be detected and its minimum detectable size.

23 Chapter 6: Low Level Design In chapter 6 the object orientated design of the Image Management Suite is discussed. The object orientated design builds on from what was discussed in the object orientated analysis and goes deeper into detail about the classes in the application. 6.1 Searching for images and eliminating duplicates The hard drive is searched to find image files. The image file types supported include JPEG, PNG and TIFF image file extensions. Once an image file is found the checksum of that image is generated. The checksum is then stored within the image to preserve the original checksum. The checksums of all image files found are compared and duplicate images are then eliminated. The file paths of each image are then stored in a file to be used in the main application itself.

24 6.2 Manual Tagging Entered Tags XML string construction <tags> <tag id= 1 > Roland Foster </tag > </tags> Embed XML within the image As shown in the above figure the user enters a tag (in this name of the tag is Roland ) the tag is then constructed into an XML string and this string is then embedded within the image s metadata using the PyExfi2 library. 6.3 Searching for images based on tags Enter Search Terms Match the search terms against the tags extracted from the images Images matching search criteria are returned As shown above the user enters the search terms, we then use PyExif2 to extract the metadata embedded within the image. We then compare the search terms with the metadata from the image to match these search terms with the tags which are currently in the image. The file paths of the images which match the search criteria are then returned to the user.

25 6.4 Automatic Tags Detection We first implement the cv. Haar - DetectObjects method to detect the objects in the image. In this case it is the detection of faces. <tag id= 0 points="(78, 83, 138, 143)">Face</tag> <tag id= 1 points="(403, 115, 471, 183)">Face</tag> <tag id= 2 points="(47, 156, 113, 222)">Face</tag> <tag id= 3 points="(177, 221, 245, 289)">Face</tag> <tag id= 4 points="(87, 239, 157, 309)">Face</tag> <tag id= 5 points="(241, 83, 311, 153)">Face</tag> <tag id= 6 points="(290, 152, 362, 224)">Face</tag> <tag id= 7 points="(313, 225, 387, 299)">Face</tag> <tag id= 8 points="(373, 240, 451, 318)">Face</tag> An XML string is then formed with all detected objects as shown above We then use PyExif2 to write the XML string as metadata within the image

26 The flow diagram above illustrates the process of automatically detecting objects, in this case the detection of faces. Firstly we implement the cv.haardetectobjects method from OpenCV to detect all faces in the image. This will provide us with the co-ordinates for each detected object. These co-ordinates are then used to form an XML string as shown in flow diagram above. Once the XML string is constructed we then use the PyExif2 module to read the metadata currently in the image and we then embed this XML string within the image.

27 Chapter 7: Testing The type of testing performed for this application was user interface testing. A total of twenty users were asked to interact with the application and to test its usability. The users were given six of tasks to perform on the interface of the application. Each user received a questionnaire in order to rate the degree of difficulty it commanded for them to complete the six tasks. Below is the questionnaire which was given to the users: Task 1: Search for images on the hard drive. Task 2: Manually tag an image There are three types of manual tagging: 1- Left click on an area in an image to tag it 2- Right click on an area in an image to add used tags 3- Off image tagging: Type in the textbox below the image to tag it. Task 3: Use the automatic tags function to automatically tag Wall clocks and Faces Task 4: Search for images via tags Task 5: Import a custom haarcascade Task 6: Edit the tags of an image by using the edit tags function

28 Rate the degree of difficulty when performing of tasks in the table below. 1 being Easy to 5 which is Difficult. Task No (Very Easy) ( Easy) (Moderate) (Difficult) (Very Difficult) Suggestions:.... Number of votes Task No. Very Easy Easy Moderate Difficult Very Difficult Table 1: Summary of the questionnaire results From the table above we can deduce that 85% of the users found Task 1, which entailed searching for images on the hard drive, to be very easily executed. For Task 2, the majority of the users (60%) found it to be easy. Task 2 had the user perform three types

29 of tagging methods. Task 3 as we can see has 40% of the users finding the task very easy and 55% of the users found it to be easy. Task 4 the majority of users found the task to be easy. In task 5 users were asked to import a custom haarcascade from the filing system. Most users found task 5 to be easy 45%. This low value indicated that it was necessary to change improve the user friendliness of task. Task 6 had 55% of the users finding it easy to perform. Additional Comments from Users The last part of the questionnaire asked users to provide any comments about the interface of the application. Most user comments were about the layout of the widgets in the application. These users suggested I rethink the layout of the widgets making them easier to locate. Two of the users suggested that the edit tags function should be available on the main screen of the application s interface. Show each tag separately which its appropriate position on the image. Degree of difficulty for all tasks Total (%) Very Easy 46% Easy 41% Moderate 12% Difficult 0% Very Difficult 1% Table 2: Average of difficulty for all tasks Table 2 shows the average degree of difficulty for the completion of all six tasks by the users. Considering that 46% counts towards these tasks being very easy to perform it shows that the application performed satisfactory overall for the six tasks but improvements should be made to strive towards making the application 100 percent very easy.

30 Percentage of ratings (%) Summary of Tasks 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 46% 41% 12% 0% 1% Very Easy Easy Moderate Difficult Very Difficult Degree of Difficulty The bar graph above illustrates the total distribution of ratings given for all six tasks by all twenty users who tested the application. We note that users who found the tasks to be Very Easy and Easy amount to 46% and 41% respectively. We also note that only one user found one specific task to be Very difficult.

31 Chapter 8: User Manual 8.1 Initial Start-up On initial start-up of Image Management Suite, the user is prompted to select his/her pictures directory in order for it to create its unique list of images. Once the user clicks OK the application will open, displaying a list of unique images from the chosen directory.

32 8.2 Viewing an image The user simply double clicks on an image to view 8.3 Tagging an image There are three types of manual tagging: 1 Co-ordinate based tagging -Left click on an area in an image to tag it 2 Used Tags - Right click on an area in an image to add used tags 3 Non-co-ordinate based tagging - Off image tagging: Type in the textbox below the image to tag it.

33 8.3.1 Co-ordinate Based tagging Click on the area you would like to tag Once the user left clicks on the area where he/she would like the tag to be placed the dialog above appears. This dialog prompts the user to enter a tag. In the figure above we can see the tag entered is hand. The user then clicks the OK button to confirm the tag or Cancel to cancel tagging the image.

34 The tag named hand has been added to the image as shown in the tags list box above Used Tags Tagging The user simply right clicks on the desired area for the tag. A dialog box containing a drop down menu with a list of used tags will then appear. The user then simply selects a tag from the list.

35 8.3.2 Non-co-ordinate Based Tagging The user simply enters in the tag in the textbox as shown on in the image above. In this case the Non-co-ordinate based tag is Happy Family. The user then clicks the add tag button to add the tag to the image.

36 8.4 Searching for an image based on tags The user can enter search terms into the text control as shown in the figure below: Figure 7: Screen shot displaying searching for an image. The user first enters the search terms (in the case the search term is hand ), then he/she presses enter to search for all images which match the search criteria.

37 Once the search button, the application returns the images which contain the tags related to the user s entered search terms. As shown in the image one image matching the search term hand was found. 8.5 Viewing Position Tags As shown in the figure above the user simply clicks on the Face to view the position of the tag in the image.

38 8.6 Automatic Tagging In order to have the application automatically detect and insert tags in images the user clicks the Auto Tags button, as highlighted by the black box, a dialog box then appears prompting the user to choose the objects to be detected. This is case the user selected Face as the object to be detected. Once the automatic tagging has been completed, the tags detected appear in the list of tags as shown in the above image.

39 8.7 Importing a haarcascade To import a haarcascade the user clicks on the Import haaracascade button. A dialog box appears prompting the user to set the file path of the haarcascade XML file, the name of the object to be detected and its minimum detectable size. The user then enters in the details in the dialog box and clicks the OK button

40 A confirmation message is then displayed to confirm that the haarcascade was successfully imported. The user can now follow the section 8.4 to make use of their newly imported haarcascade. 8.8 Editing tags The user double clicks on the tag (in this case it is a Face ) to open the editing box for that specific as shown in the figure above.

41 The user then enters in the new name for the tag, He/She then clicks on the Save Changes button located under the scrollable list widget to save the tags (as shown above).

42 8.9 Deleting tags To delete a tag the user simply clicks the checkbox of the desired tag to be deleted (in this case the tag is named Face ) He/She then clicks on the Save Changes button located under the scrollable list widget to delete the currently selected tag (as shown above). In order the delete multiple tags the simply clicks all checkboxes of the tags which he/she would like to delete and click the Save Changes button when finished.

43 Chapter 9: Code Documentation The documentation of the code has been fully documented by inserting comments each class and each method. It is too long to include in this document but the final source code with all its documented code will be stored on a CD and placed in an envelope.

44 Conclusion A detailed description of the problem is stated as well as the software solution to the problem in Chapter 2 of this document. The user requires an easy-to-use, interactive application. The application finds all images and eliminates duplicate images. It also allows the users to label images with tags. The problem stated is that users had a tough time tagging every image and the implementation of automatic tagging makes it easier for the user to manage this tagging process. Retrieving of images is also an easy task by simply entering the search terms based on the tags in these images. The final application quite clearly meets the necessary requirements to create an application which easily organises and retrieves images as well as their metadata.

45 References [1] Kaehler, G. B. (2008). Learning OpenCV. USA. [2] Olivier Tilloy, Pyexiv2 Documentation. Available at: [3] Willow Garage, Object Detection. Available at: bjdetect_cascade_classification.html#haardetectobjects. [4] Kaehler, G. B. (2008). Learning OpenCV. USA. [5] wxpython, wxpython. (2012).