System Reference 2013

System Reference 2013

Content List of graphics...3 List of tables...4 1 Purpose of LynxSight...5 2 Setting up project...6 2.1 Project dependences...6 2.1.1 Operating system...6 2.1.2 Eclipse IDE...6 2.1.3 Other development enviroments...7 2.2 Setting up IDE...7 2.2.1 Tegra android development pack...7 2.2.2 Integrating Eclipse IDE...8 2.2.3 Android Development Tools...8 2.2.4 Android NDK...9 2.2.5 Tesseract for Android...9 2.2.6 OpenCV for Android... 10 2.2.7 Android support v7... 11 3 Design reqirements... 13 4 Application workflow... 14 5 Description of packages... 15 6 Description of classes... 16

List of graphics Pic 1 Example of Android.mk file used in project LynxSight... 11 3

List of tables Table 1: Description of application packages... 15 Table 2: Description of application classes... 16 4

1 Purpose of LynxSight LynxSight is mobile application built for Android OS devices. It is an OCR system designed for use by visually impaired people. LynxSight has following features: Voice commands Simple navigation Text reading (text to speech) LynxSight allows user to take picture of text anywhere he is read it aloud. Text on Picture is recognized and higlighed what is done by decompozition module. Application is designed in Java language using android native libraries such as OpenCV or Tesseract. 5

2 Setting up project Sources and resources for project LynxSight are organized in Eclipse Android projects. Eclipse is used because of availability and suitability of tools used in developing of project. Setting up Eclipse IDE is capitol 2.2. Besides setting up the IDE Android development tools ADT and NDK for native development are presented. In this section are also presented used third-party libraries such Tesseract OCR tool and OpenCV library to increase of officiency of manipulating with rastr-based images. Finally, requirement of the majority of Android-based devices compatibility should be met. Therefore Android support modul is integrated to the project to ensure smoothly run of application on devices with older version of OS Android. Some of mentioned development tools are integrated to Tegra Android development pack presented by NVIDIA. Usage of this pack is highly recommended for unexperienced, but also becouse of development efficiency experienced developers too. 2.1 Project dependences 2.1.1 Operating system Windows XP (32-bit), Vista (32- or 64-bit), or Windows 7 (32- or 64-bit) Mac OS X 10.5.8 or later (x86 only) Linux (tested on Ubuntu Linux, Lucid Lynx) o GNU C Library (glibc) 2.7 or later is required. o On Ubuntu Linux, version 8.04 or later is required. o 64-bit distributions must be capable of running 32-bit applications. 2.1.2 Eclipse IDE Eclipse 3.6.2 (Helios) or greater Note: Eclipse 3.5 (Galileo) is no longer supported with the latest version of ADT. Eclipse JDT plugin (included in most Eclipse IDE packages) JDK 6 (JRE alone is not sufficient) Android Development Tools plugin (recommended) Not compatible with GNU Compiler for Java (gcj) 6

2.1.3 Other development enviroments JDK 6 (JRE alone is not sufficient) NDK For all development platforms, GNU Make 3.81 or later is required. Earlier versions of GNU Make might work but have not been tested. A recent version of awk (either GNU Awk or Nawk) is also required. For Windows, Cygwin 1.7 or higher is required. The NDK will not work with Cygwin 1.5 installations. Apache Ant 1.8 or later Not compatible with Gnu Compiler for Java (gcj) Note: Some Linux distributions may include JDK 1.4 or Gnu Compiler for Java, both of which are not supported for Android development. 2.2 Setting up IDE There are two approaches to properly set up development enviroment for running project LynxSight. First is download and install all development tools separatelly and avoid installing development tools which could be useless. Second approach expects download and installation of Tegra android development pack. This option si recommended for developers less experienced in android development and also every developer which has not configured his enviroment yet. Consider properly which approach would you choose before started developing. 2.2.1 Tegra android development pack Tegra android development pack includes all software tools required to develop for Android platform. The pack is availiable to download at https://developer.nvidia.com/tegra-android-development-pack. Before download registration at NVIDIA developer zone and signing up is required. To be able to get the pack you have to apply Tegra registered developer program. After application it is necessary to wait until your request is approved. After installation file is run you are forced to choose which tools you want to download and install. Select Eclipse IDE Android Software development kit (SDK including ADT) Android Native development kit (NDK) Cygwin (when using Windows OS) Apache Ant 7

Java development kit (JDK) OpenCV for Android platform Use of current version of development tools is highly recommended option. Download and installation can take even several hours according to local internet connection and variety of selected Android SDK versions. It is recommended to install tools for every version since Android API 7. Notice where are tools installed and properly set system enviroment variables to Cygwin, Android SDK, JDK, NDK and Apache ant. To set system enviroment variables go to Control panel System Advanced system settings Enviroment variables. Edit Path variable with addition of Path of mentioned tools above if they have not been set yet. 2.2.2 Integrating Eclipse IDE Different status of developer means different configuration. If you are using Tegra android development pack, your Eclipse IDE should be binded with Android SDK and NDK. If it is not, or different approach is used, including building enviroment from scratch or adding tools to existing Eclipse enviroment, several steps should be performed. To import LynxSight project to Eclipse IDE go to File Import Existing Projects into Workspace, select Browse and find destination of Eclipse project LynxSight in file system. Project appears in local Workspace. Probably it would contains errors and additional configuration is required. 2.2.3 Android Development Tools In Tegra version check if Android SDK is properly set. Android SDK is core of Android Development Tools. Android development tools provide tools to compile, debug, run and test android applications. Also Android device emulator is included. At Windows Preferences Android check SDK Location which should match the location of Android SDK in file system. In non Tegra version when Android SDK is not present, install perform the following steps: 1. Start Eclipse, then select Help > Install New Software. 2. Click Add, in the top-right corner. 8

3. In the Add Repository dialog that appears, enter "ADT Plugin" for the Name and the following URL for the Location: https://dl-ssl.google.com/android/eclipse/ 4. Click OK. If you have trouble acquiring the plugin, try using "http" in the Location URL, instead of "https" (https is preferred for security reasons). 5. In the Available Software dialog, select the checkbox next to Developer Tools and click Next. 6. In the next window, you'll see a list of the tools to be downloaded. Click Next. 7. Read and accept the license agreements, then click Finish. If you get a security warning saying that the authenticity or validity of the software can't be established, click OK. 8. When the installation completes, restart Eclipse. 2.2.4 Android NDK In Tegra version check if Android Native Developmen Kit (NDK) Path is properly set. The NDK is a toolset that allows you to implement parts of your app using nativecode languages such as C and C++. At Windows Preferences Android - NDK check the NDK Location which should match the location of Android NDK in file system. In project LynxSight native modul Tesseract and OpenCV for Android are used. In non Tegra version download Android NDK from http://developer.android.com/tools/sdk/ndk/index.html and unzip to the file system. Then set the NDK path as was described above. If project was imported incorrectly, or there is need for creating new project and importing source and resource files manually, follow the following steps. Right click to the project Android tools Add native support. Check if jni directory was generated. 2.2.5 Tesseract for Android Project LynxSight already contains tess-two.jar library in libs directory. It also contains required.so files liblept and libtess in libs/armeabi and libs/armeabi-v7a folders. If you build your project using Windows you may use this compilated libraries. However, if you build your project in another operating system you should build Tesseract engine by yourself. Download the source or clone this git repository at https://github.com/rmtheis/tess-two. This project contains tools for compiling the 9

Tesseract, Leptonica, and JPEG libraries for use on Android. It contains an Eclipse Android library project that provides a Java API for accessing natively-compiled Tesseract and Leptonica APIs. Go to your Android.mk file and add the following lines between include $(CLEAR_VARS) and include $(BUILD_SHARED_LIBRARY: APP_PLATFORM := android-7 Use command line to build Tesseract 3.02 toolset for Android. Set current directory to directory where you downloaded tess-two project. Now you are going to build tesseract for your OS enviroment. Make sure you have set enviroment variables to android NDK, android SDK and Apache ant. Variable --path means path to your tesstwo project. cd <project-directory>/tess-two ndk-build android update project --path. ant release After succesfull build of tesseract project import tesseract project to your Workspace. Then right click to the project Tesseract Properties Android Check Is library radio button. After that copy tess-two.jar library from Tesseract project build folder to libs folder of your project. Also copy content of Tesseract libs folder to your libs folder. It contains required native.so libraries for tesseract and leptonica support. Then right click to your project Android tools Fix Project properties to make sure tesseract tools are ready to use. After the initial import, on a non-windows (Linux and Mac OS) operating system Eclipse will still show build errors for applications with native C++ code. To resolce the issues, please do the following: Open Project Properties -> C/C++ Build, and replace Build command text to "${NDKROOT}/ndk-build" (remove.cmd at the end). 2.2.6 OpenCV for Android OpenCV is toolset for image processing. It is important element of LynxSight project becouse of presence of Decomposition modul. OpenCV for Android toolset is also integrated in Tegra android development pack. Once using Tegra android development kit, OpenCV toolset project should be installed. Find the project in NVIDIA pack and inport to the Workspace. Reference your project to project of OpenCV or reference to it s library. To reference to OpenCV project right click to the project OpenCV Properties Android Check Is library radio button and then right click to your project Properties Android Add and select 10

library project OpenCV for Android. Make also sure, that OpenCV is placed in the same directory. Another way, which is more common is to import OpenCV.jar library to your project. Once the OpenCV project is labeled as Library project, opencv library 2.4.5.jar is placed in its bin folder. Copy this library to the libs folder of your project. The same way is Tesseract for Android imported. If you are not using Tegra android development pack, download OpenCV for android SDK at http://sourceforge.net/projects/opencvlibrary/files/opencv-android/. Unzip the pack into the particular folder and notice the path. Follow the instructions like Tegra pack user described above. Go to your Android.mk file and add the following lines between include $(CLEAR_VARS) and include $(BUILD_SHARED_LIBRARY: OPENCV_CAMERA_MODULES:=on OPENCV_INSTALL_MODULES:=on An example is shown on Pic. 1. Pic 1 Example of Android.mk file used in project LynxSight For more information abou setting up the OpenCV for Android toolset follow the instructions at: http://docs.opencv.org/doc/tutorials/introduction/android_binary_package/o4a_s DK.html#get-the-opencv4android-sdk 2.2.7 Android support v7 Although there is a presumption, that software developed with official android development tools should be able to run at most android based devices, the whole system involve and older version use not to respect new features. That is why Android 11

support v7 is used. In project LynxSight it brings compatible action bar to all Android versions since Android SDK version 7. To get Android support v7 library follow the following instructions: 1. Start the Android SDK Manager. 2. In the SDK Manager window, scroll to the end of the Packages list, find the Extras folder and, if necessary, expand to show its contents. 3. Select the Android Support Library item. Note: If you're developing with Android Studio, select and install the Android Support Repository item instead. 4. Click the Install packages... button. Fill metadata in uses-sdk tab in your Android manifest file. MinSdkVersion field is required. <uses-sdk android:minsdkversion="7" android:targetsdkversion="17" /> After installing the particular packages in your AVD manager notice their location and import project Android support v7 appcompat to your Workspace. Then put a reference from your project to the Android support project. It is important to use project reference instead of library reference becouse resource files in Android support project. Putting a project reference process is described in OpenCV and also Tesseract section. For troubleshooting and more details about Android support v7 setup follow http://developer.android.com/tools/support-library/setup.html 12

3 Design reqirements Designed of application has to follow some requirements: o User flexibility o Highest market target User flexiblity means that applications should be usable without any special conditions. For example application should be usable without access to internet. This requirement determines that all application components should work offline. Highest market targeting means that application should be programmed in way that maximizes number of its potential users. Application is therefore implemented on Android OS for versions up to 2.1. 13

4 Application workflow Application workflow starts with Main menu displayed. User has choice to go to settings menu or to Picture capture screen. Settings menu allows user to set som basic settings sucha s voice assistance, language of OCR and text to speech and so forth. Picture capture screen is first step to OCR. In this step camera is initialized and camera preview is displayed. User has possiblity to change flashlight settings. Picture is taken simply by pressing any point on preview screen. Focus is done automaticaly after the touch. After the Picture is taken decomposition task is fired. In this step application tries to find logical text regions. When task is finished result screen is displayed with highlighted regions of text. User has possibility to read selected region aloud simply by touching on it. The touch on certain region fires OCR task which results in recognized text from region that is read aloud by text to speech module. Main menu Settings Capture screen Result screen 14

5 Description of packages Table 1: Description of application packages Package Sk.LynxSight Sk.LynxSight.camera Sk.lynxsight.OCR Sk.lynxsight.storage Sk.lynxsight.magnifying Fejero.lsight.decomposition Fejero.lsight.font Description This package contain basic classes of application logic. Main activity class be found here along with classes covering general functionality, This package collects classes of camera preview functionality. The OCR is done using classes of this package. OCR requests are done using asynchronous tasks using native code. Classes for storing information about application state and settings are stored here. For example SQLite Access layer. Classes of camera for Magnifying glass feature This package covers classes which brings tools for decomposition of document taken by digital camera This package include classes which bring text size detection tools using opencv library 15

6 Description of classes Table 2: Description of application classes Package Class Description Sk.LynxSight MainActivity MainActivity represents general structure of application logic. Activity layouts and their contents, actions and behavior is programmed here. We use only one activity in LynxSight for all submenues so far. Logger The Logger class provides loging functionality mostly for debuging purpouses. This is just stub. Orientation Enum for display orientation naming used in application ScreenDisplayed SensorListener SettingsManager Utils Enum for possible menus (layout) currently displayed on screen. Event listener for listening on screen orientation changes in three axes. This is a container class that groups and stores all user settings data. Static class for general purposes methods. Sk.lynxsight.camera CameraPreview The class represents camera 16

preview view displayed on camera screen when taking Picture. Camera settings and preview quality is managed in this class. Sk.lynxsight.OCR OCRAsyncTask The Class OCRAsyncTask. This class provides OCR method to get string representation of bitmap. We use tesseract 3.01 engine encapsulated in Java interface. It is important to initialize tessdata from local database. Tessdata file have to be in particular folder. In this case it is root/tesseract. OCRBundle The Class OCRBundle. Encapsulates language of OCR and bitmap to be translated. This bundle is important, becouse asynchronus call get only one reference. OCRCorrection This class performs some post processing corrections on recognized texts in order to make results better, Sk.Lynxsight.storage DatabaseHandler The class is Access layer to persistent data storage. In current implementation it is SQLite handler. 17

Sk.lynxsight.magnifying MagnifyCameraPreview Interface and camera holder for Magnifier tool. This class has simmilarities with CameraPreview class. MagnifyingActivity Activity class which activate camera-magnifying glass tool. This activity initialize camera preview. Fejero.lsight.decomposition Area BlockFinder Decomposition Threshold Tile On of basic elements of Decomposition algorithm. Contains all necessary information about individal text block in docment sch a size and position in document. In this class is algorithm for text detecting implemented. Graphical Seed-Fill algorithm is used combinated with random selection of tiles. Interface for LynxSight application, which provides call for list of detected text blocks. Class which provides canny tools for binary image output. Uses OpenCV for android graphic library. The basic element of decomposition algorithm. Provides information about 18

size, location, textual content and calculate those data for himself. TileDrawer TileManager Class provides method which draws tiles in image view. Usefull in debug mode. This class cuts the bitmap into a 2 nd array of tiles and analyse those tiles. Also calculates number of rows and cols and size of one tile. Also creates lists of every group of neightbour tiles and generate Areas. Fejero.lsight.font FontSizeDeterminer Class calculate average height of characters sed in text. From this data is appropriate size of tile calclated. 19