Summary. Introduction... 1 Troubles found and solution propose... 1 Architecture... 3 Tests... 8 References... 15

Transcription

1 MrCirrus Azure. Map-Reduce in high level bioinfo clouds (v 1.0) Óscar Torreño Tirado Daniel Luis Ramet Barea Coordinators: Oswaldo Trelles, Johan Karlsson University of Málaga Computer Architecture Dept. Málaga, 29 July 2011

2 Summary Introduction... 1 Troubles found and solution propose... 1 Architecture... 3 Tests... 8 References... 15

3 Introduction The problem that arises and comes to programming in the Cloud is the ability to apply the philosophy Map-Reduce for parallel execution of sequential programs. Another problem found are old programs (legacy applications) where priority is given to reuse them. Therefore, the solution to this problem is to enable an environment that allows execution of sequential programs in the field of bioinformatics, as well as the ability to reuse applications where you only have the executable. Now, we present a possible solution and to contrast with other possible settings. We will use Microsoft Azure platform service, commonly called "Cloud". Then we will detail how to work in the Microsoft cloud. Basically, Azure Web Role working through, this acts as a front-end user, and Worker Role, which background processes are executed while the application is active. Communication between all elements is performed by the data store that provides Microsoft. This store is composed of three elements: blobs, queues and tables. For a detailed explanation of application development can be seen the use for each element. Troubles found and solution propose First, we ll do a summary with the way to work with Microsoft cloud in order to explain the troubles that we have found. The basic data must be in the cloud (i.e. exe programs and sequences), this is the first trouble that we had found, how to upload and download files from/to the cloud. And maybe this files could be very large. We solve this trouble developing the project ManagementFiles in C# (project will be explained in next chapter). Each worker that executes a program (i.e. dotplot, blast,etc.) needs as first step to download all its needed files from the cloud and as second step upload the results. This is another trouble that we had found, how to execute in the Cloud a program written in C or any program language. We solved this compiling the program with the compiler Minimalist GNU for Windows (MinGW). Also, we have created Thread inside the worker execution whose work is executing the program with the specified arguments. Another problem is that the execution must be at the local way so we need to download all the needed files, do the work and then upload the results to the blob container. Related with the virtual machine size where are located each group of workers, could be of different sizes: - The Extra Large size contains 8 processors. So if we put 8 workers (according to the price that we want to pay, we can have more or less workers) we have 64 processors to give them work. 1

4 - The Extra Small size contains 1 processor. Actually we are using this size because we don t have the account without constraints yet. Now we are going to see the procedure to execute MapReduce (at the last chapter there are some samples): DOTPLOT or Fragments First we need to execute the MAP program. So previously we should have uploaded the program and the sequences that we want to work with them. Once we have executed MAP, we need to upload the results because of the limitation to execute with local files. We have made automatic the upload of the parts of the sequences and also the two scripts (script-map and script-reduce). After that, it s time to execute the map script. Here, we have found another problem, if we give Azure the script it will run sequentially (in case of that the worker has the files locally), so to solve that we have created a class that read the script per lines and execute each line in a different worker (which download the files that need to execute locally). When all the workers end their job, we can execute Reduce program to merge the results from last step. This is another trouble that we had found, because unless this script has one line, we need to download the needed files in order to execute the program. So to solve that we had created a class that downloads the files, executes the program and uploads the results. BLAST At the moment, it s in execution process. It has to install some extern libraries and it has to do for each virtual machine. 2

5 Architecture Then we show a packages diagram with the architecture of the system. Figure 1: System architecture. Firstly we have Web_Role package that represents the system front end. A web page where we can do execution requires and also the download/upload of files. Then we show two captures with the two tabs of the page. 3

6 Figure 2: MapReduce Tab (execution) Figure 3: Download/Upload tab. 4

7 Secondly, we have five packages which the Web_Role communicates throughout a message queue. - Map Worker Role Figure 4: Class associated with the map worker role. - Script Worker Role: Class that read the script per lines and assign job between the MrCirrus Worker Roles. Figure 5: Class associated with the script worker role. 5

8 - MrCirrus Worker Role: This is the main Class, contains the hard job. - Red Worker Role Figure 6: Class that execute a line from the script. Figure 7: Class that executes the reduce script. 6

9 - Files Worker Role Figure 8: Class that manages the upload/download of files. To finish we have the package ManagementFiles, which is used by quite all the another packages. Then we show a class diagram of this package. Figure 9: Classes to manage the upload/download of files. 7

10 Tests First we want to show how works the download/upload tab. - List all the documents at specified container: You must type the name of the container and click at List Blob button. Figure 10: Result of list documents container. - List local files: You must click on List Local button. Figure 11: Results of list local. 8

11 - Download a file: after list a concret container, you can type the name of a file that you want to download and click on download button. Figure 12: Results of download a file. - Upload a file: you must type the full path of the file that you want to upload and click on upload. Figure 13: Results of upload a file. 9

12 Now we show a full execution sample from map program to reduce program. In order to the workers download and upload the files automatically, we have established some marks at this way. The marks are these: Mark Meaning $ Download in Map Worker Role * Combine local path with the name of the file in Map Worker Role # Download in MrCirrus Worker Role & Upload in MrCirrus Worker Role First we need to upload the needed files to execute, in our chase, two sequences (the first one is a part of Buchnera APS and the second one Buchnera BPS, we didn t have uploaded the full sequence because of the limitation of the free account), the map program, fragments and reduce programs. Once we have uploaded the files to the container: Figure 14: State of the container after upload the files. After we have uploaded the map program and the sequences, we can run the map program that causes this interaction: 10

13 Figure 15: Interaction diagram caused by the execution of map program. The call has been this: $mapafv2.exe AllFragv3.exe $SeqX1.seq $SeqY1.seq 3 2 &AF.bin *AF.script 100 RedAFv3.exe. After the execution we can see the generated files listing again the container. 11

14 Figure 16: State of the container after map execution. We see that we have two scripts and sequences parts. Now we have to execute the map script with the calls to fragments program. To make that we must type the name of the script and click at execute script button. This action causes this interaction. 12

15 Figure 17: Interaction diagram of script execution. After the script execution we can lit again the container and check that all the results are generated. 13

16 Figure 18: State of the container after the script excution. Last step is execute the reduce script, in order to do that we must type the name of the script and click at execute reduce button. Once we have done all these steps, we can check if the result file, AF.bin in this chase, has been generated. Figure 19: State of the container after final step. 14

17 References azurestorageexplorer.codeplex.com/ 15