Windows Azure: Cloud Reader

Transcription

1 Windows Azure: Cloud Reader Design Document Project Number SDDEC10-09 Client Microsoft Corporation Shawn Hanson Dan Leeaphon Matt Thalman Faculty Advisors Dr. Suraj C. Kothari

2 Table of Contents 1 Executive Summary Definitions Problem Statement Solution Approach Operating Environment...4 Windows & SQL Azure...4 RIA Services...4 Silverlight NET Framework Intended Users & Uses Assumptions & Limitations Expected End Product & Other Deliverables Design Objectives Functional Requirements Design Constraints Technical Approach Considerations & Results...6 State Changes...6 Data Binding...7 User Control Customization...7 Third Party Service Libraries Testing Approach Considerations Detailed Design...7 Screen Flow Narrative...7 Screenshot 1:...8 Screenshot 2:...8 Screenshot 3:...9 Screenshot 4:...11 RIA Services...12 Internal Navigation...12 Scheduler...12 Throttle...12 Queue...12 Search Agent Resource Requirements...13 Hosting Project Schedule Closing Summary Project Team Information Appendices...16 Figure Figure Figure Figure Figure

3 1 Executive Summary The objective of this project is to build a service which can sufficiently test the ability to scale an application on the Azure platform as well as discover migration issues when migrating from a traditional server side application to having the application hosted on the Windows Azure platform. The project and documentation will be completed by December 2010 with any costs or licenses covered by the Microsoft Development Team. 1.1 Definitions Search Agent: An abstraction of a real news data source such as the free Twitter or Digg APIs, or premium data API s where the user has an account, or a custom built web crawler that scrapes information from a defined domain. It is the Data Source's job to return formatted Result Sets to the RDBMS based upon the Rule that it was assigned. Label: Defines a grouping of Rules and is associated with one Result Set. A Label is used to define the search context. Rule: Defines an association between one Data Source and one or more Filters. Filter: Defines how Results shall be fetched from a Data Source. An example is a Date Filter, which defines a lower bound on how recent a Result from a Data Source must be. Another example could be a Keyword Filter which defines a keyword search string consisting of search operators such as AND, OR, EXACT, or NOT. Results: Individual Results returned by a Data Source. The Search Agents interacts with the RDBMS to store Results for the User to later request and view on demand. Search Session: Search Agents store meta data concerning the status of processing an instance of a Rule in a Search Session. Prototype: Working copy that is in bare-bones form. 2 Problem Statement The Windows Azure platform offers developers the power and flexibility of cloud computing at a price tag that scales with respects to the resources consumed by applications. With every new system, there are bugs that need to be discovered and issues that must be uncovered before the system can become an industry standard, and the Windows Azure platform is no exception. It is our goal to play the role of a third party developer migrating a traditionally hosted application to the Azure cloud while documenting the issues and solutions we encounter. 2.1 Solution Approach Traditional search does an adequate job at keyword based searches, but lacks the awareness of the context of the search. Our approach gives the user more finesse by allowing them to specify the context of the search. A good analogy is a news reader application that the user can configure to seek out

4 interesting content. Search is a good application to test the scalability Windows Azure platform because providing contextual searches for users can require large amounts of storage and computing. To implement the application we will be integrating a Silverlight client with a RIA WCF middle-tier that will act as the component that coordinates communication with the Windows Azure platform. The process of integrating the system will help us accomplish our goal of gleaning insight into issues that may arise during the development lifecycle on the Azure platform. In addition, we shall iteratively implement an auditing system to capture scalability metrics to analyze the performance of the platform. 3 Operating Environment Windows & SQL Azure "Windows Azure is a cloud services operating system that serves as the development, service hosting and service management environment for the Windows Azure platform. Windows Azure provides developers with on-demand compute and storage to host, scale, and manage web applications on the Internet through Microsoft data centers. Windows Azure is a flexible platform that supports multiple languages and integrates with your existing on-premises environment." For more information about Windows Azure please visit: windowsazure/ "Microsoft SQL Azure Database is a cloud-based relational database service built on SQL Server technologies. It provides a highly available, scalable, multi-tenant database service hosted by Microsoft in the cloud. SQL Azure Database helps to ease provisioning and deployment of multiple databases. Developers do not have to install, setup, patch or manage any software. High availability and fault tolerance is built-in and no physical administration is required. SQL Azure Database supports Transact- SQL (T-SQL). Customers can use existing knowledge in T-SQL development and a familiar relational data model for symmetry with existing on-premises databases. SQL Azure Database can help reduce costs by integrating with existing toolsets and providing symmetry with on-premises and cloud databases. " For more information about Windows & SQL Azure please visit: windowsazure/sqlazure/ RIA Services "Microsoft WCF RIA Services simplifies the traditional n-tier application pattern by bringing together the ASP.NET and Silverlight platforms. RIA Services provides a pattern to write application logic that runs on the mid-tier and controls access to data for queries, changes and custom operations. It also provides end-to-end support for common tasks such as data validation, authentication and roles by integrating with Silverlight components on the client and ASP.NET on the mid-tier." For more information about RIA Services please visit: Silverlight 3 "Silverlight is a powerful development platform for creating engaging, interactive user experiences for Web, desktop, and mobile applications when online or offline. Silverlight is a free plug-in, powered by

5 the.net framework and compatible with multiple browsers, devices and operating systems, bringing a new level of interactivity wherever the Web works." For more information about Silverlight please visit: Framework 3.5 "The.NET Framework is an integral Windows component that supports building and running the next generation of applications and XML Web services. The.NET Framework is designed to fulfill the following objectives: To provide a consistent object-oriented programming environment whether object code is stored and executed locally, executed locally but Internet-distributed, or executed remotely. To provide a code-execution environment that minimizes software deployment and versioning conflicts. To provide a code-execution environment that promotes safe execution of code, including code created by an unknown or semi-trusted third party. To provide a code-execution environment that eliminates the performance problems of scripted or interpreted environments. To make the developer experience consistent across widely varying types of applications, such as Windows-based applications and Web-based applications. To build all communication on industry standards to ensure that code based on the.net Framework can integrate with any other code." For more information about the.net Framework please visit: w0x726c2(vs.90).aspx 4 Intended Users & Uses Users Microsoft User Uses Shall glean insight into the migration process from a traditionally hosted application to the Windows Azure platform. Shall search for and consume relevant content. 5 Assumptions & Limitations Assumptions The cloud will give us scalable resources and have adequate uptime. The User will have Silverlight 3 installed. 3rd party data sources will have sufficient uptime. Limitations Although resources are scalable, we need to be able to throttle how much is used. Although 3rd party data sources may have sufficient uptime, we must still take care not to violate 3rd party TOU agreements. 6 Expected End Product & Other Deliverables

6 6.1.1 Design Objectives The main objective of the project is to create an application that tests Windows Azure's capabilities and general flaws. This includes: 1. Testing and being able to control the cloud's hosting power 2. Contain and provide personalized information 3. Accomplishing Azure integration with the application To meet the design objectives, we chose to do a personalized article (Google) reader. With this project design we will accomplish all of the design objectives. First, the project will have to throttle our background operations (fetching personalized material) in the cloud to avoid large processing bills each month. Second, a personalized reader is only as good as its personalized content, to the project is very user centered. Finally, the project could be hosted on a personal server but will be uploaded to the Azure Cloud for full functionality Functional Requirements Must be hosted on the Azure Platform Must use Silverlight for the user view Must use RIA Services to perform the middle tier operations No user may interfere with any other user (security). Must be a user centric applications which includes user profiles Design Constraints There are some limitations involved with using third party libraries, most notably rate limiting issues. To avoid issues with banning as result of exceeding the set rate limits we will use third party libraries which adhere to rate limits internally. Our service must be run on the Microsoft Azure Cloud and must support Silverlight. This means that the client must also support Silverlight which at the time is not supported by Linux, Unix, *nix systems. The application binary runs in the cloud so it must be accessed remotely by the client which must also have an Internet connection Technical Approach Considerations & Results State Changes A VisualStateManager is used to update the state of the Home page and login button. So when a user logs in, the sidebar is shown with the list of the user's labels, and the login button is changed a personalized welcome message and a logout button. For more information about State Changes visit: system.windows.visualstatemanager(vs.95).aspx

7 Data Binding Data binding is used to populate each user's list of labels. Data binding is a way to keep the database in sync with the Listbox control. For more information about Data Binding visit: User Control Customization The list of labels are held in a Listbox. The Listbox is customized so that it holds a list of Stackpanels that allow a label to be paired up with a remove button. For more information about User Control Customizations visit: silverlight-tutorial-part-5-using-the-listbox-and-databinding-to-display-list-data.aspx Third Party Service Libraries We will use third party libraries to interface with the various services we intend to search (Twitter, Digg, Reddit, custom web crawlers, etc.) in order to be compliant with third party service restrictions such as rate limiting and query issues. These libraries will also help us quickly adapt has third party service interfaces change because we will simply have to update to the updated library which changes with the third party interface updates Testing Approach Considerations We will approach testing from two categories, automated testing and visual inspections. Automated testing will be utilized wherever possible in order to allow for regression testing. Automated testing will consist of code based unit tests that range from basic CRUD (create, retrieve, update, delete) operations of entities, to stress tests that will test the ability of the system to throttle resource utilization up and down as the system experiences various user loads. Separate tests will be made to test individual Search Agents for their ability to retrieve relevant results from various data sources using supported Filters. Visual inspections will be made pertaining to the operation of the Silverlight interface. In addition to visual and unit testing, all major milestones will be subject to peer review. 6.2 Detailed Design Screen Flow Narrative Please refer to Appendices Figure 3 for this subsection. Screenshots 2-4 assume the user is a returning user with at least one label already created.

8 Screenshot 1: 1. Open up the CloudReader page. 2. If you are an existing user AND signed in, it will show the list of Labels you have already created under the My Label heading. Otherwise you will need to register and there will be no Labels in that list. 3. Initially, no Label will be highlighted. This means that the Edit Selected Label button will be inactive and grayed out and there will also be nothing shown in the frame to the right of the list. Screenshot 2:

9 1. Open up the CloudReader page. 2. Click on a Label 3. The frame to the right will populate with Articles related to the Label that the user has defined. The Edit Selected Label button will be enabled. Screenshot 3:

10 1. Open up the CloudReader page. 2. Click on the Add a Label button. 3. A Label called New Label will appear without the Remove link next to it. 4. A form will populate the frame on the right that will prompt the user for details of the new label.

11 Screenshot 4: 1. Open up the CloudReader page. 2. Click on a Label. 3. Click the Edit Selected Label button. 4. A form will populate the right frame with the same form as when creating a new Label. However, this form should be pre-populated with the current information of the selected Label. The forms as shown in the 3rd and 4th screenshot have the following behaviors: 1. An error message will notify the user if the user has failed to name the Label, define at least one Data source, and define at least one Filter for each Data source. 2. The Save button will not be enabled until the required areas have been filled out. 3. The user will not be able to go over their credit limit when creating the Label. 4. The Filter area will not be enabled until a Data source has been selected. This way the user can define and see that there can be multiple Filters for a single Data source. 5. The search box at the bottom will search for those words that also fit your Data source and Filter criteria. At this time, an article will only be pulled if it is within a correct Data source, matches the Filter criteria, and has all of the words defined in the search box.

12 RIA Services RIA Services is a middle-tier layer that coordinates client-side calls to the server by implementing the View-Model design pattern. This enables the system to maintain decoupling between presentation and business logic. The decoupling is implemented in the domain service called CloudReaderDomainService. The service exposes CRUD operations upon business models through the use of the presentation model design pattern. The presentation model design pattern requires the client to perform CRUD operations upon a proxy model rather than the actual business models. The CRUD operation logic is encapsulated within the proxy models. This second layer of decoupling frees the client from having to worry about domain specific logic when coding from the client perspective. The business models can be viewed in Figure 4. Internal Navigation The navigation within the program is being treated in such a way that if a user was authorized to view their information, they are able to regenerate any page they visit by pasting a copy of their URL. This is a important feature of the design for the development team in that it allows for direct site testing. Having the full URL allows the tester to view a site directly without having to go through a sequence of events. For the user, this is beneficial in a couple ways. The user would be allowed to bookmark a particular URL and directly visit that site. If the user was not authorized to view the information, he/she would be asked to log into the application. Once authorization has been achieved, the page would load like normal. When a user wants to duplicate a website, such as copying the URL to another tab in their browser, he/she will not have to log in and perform the necessary actions to go back to the same state that the original site was in. Scheduler The Scheduler plays the role of a writer in the classical readers and writers computing problem. Its responsibility is to query the RDBMS for Rules that have not been processed within some specified amount of time. Further testing will be needed to determine what the appropriate period between each session is, but for prototyping an arbitrary value will be chosen. Once the Scheduler has obtained a set of Rules from the RDBMS it puts the Rules into the appropriate Queues by Search Agent type. Throttle The Throttle is a background thread that periodically checks the various Queues to ensure that the approximate wait time of any Rule is kept within reasonable thresholds. However, further testing with the prototyped system will be needed in order determine the appropriate thresholds. When a Queue is found to be exceeding its threshold the Throttle will balance the load by instantiating another Search Agent thread of the appropriate type. If on subsequent checks the Throttle finds that the wait time for a Queue is within reasonable thresholds, then the Throttle will kill off any extra Search Agents of the appropriate type. Queue The Queue will be a decorator that wraps around the Windows Azure Queue Service in order to provide additional business logic that Search Agents, the Scheduler, and the Throttle will use to perform their duties. The decision to rely on the Windows Azure Queue Service rather than a traditional data structure was based upon economic considerations. The cost per compute hour on the Windows Azure platform is

13 $0.12/hr while the cost per 10K transactions using the Queue Service is $0.01. Since each Queue is a shared resource their utilization rates will be high, and this design decision will significantly reduce operation costs. Search Agent A Search Agent plays the role of a reader in the classical readers and writers computing problem. A Search Agent consumes a Rule from a Queue of the same type and performs queries on the appropriate data source based upon the semantics of the Rule. As a Search Agent receives results from its queries it formats that information info Result entities and stores them in the RDBMS for later retrieval by the Silverlight client. 6.3 Resource Requirements Please be aware that the following are only estimated costs based upon the Windows Azure Platform TCO Analysis Calculator during the first year of live release. During development hosting will be provided by Microsoft.? Hosting Windows Azure Usage Rate Initial Per Month Computing 730 hours/month $0.12 /hour $87.60 $1,051 Storage (unstructured) Storage transactions 0.5 GB $0.15 /GB/month $0.08 $1 5 10K tx/ month SQL Azure Usage Rate $0.01 /10K tx $0.05 $1 Initial Per Month 1GB databases 0 units $9.99 /month $0.00 $0 10GB databases 1 units $99.99 /month $99.99 $1200 Bandwidth Usage Rate Inbound Outbound Project Schedule avg. GB/ hour avg. GB/ hour Initial Per Month $0.10 /GB month $ $2,190 $0.15 /GB month $ $6,570 Total $ $11,013 Initial Per Year Initial Per Year Initial Per Year Milestone Deliverables Date Estimate

14 1. The "Creating a Label", "Saving a Label", and "Editing a Label" use cases shall be prototyped. 2. The Scheduling Rules, Rule Based Searching, "View Results", "Remove Results", and "Remove Label" use cases shall be prototyped. Deliverable 1 the Labels Menu shall be prototyped. Deliverable 2 the Edit Label Form shall be prototyped. Deliverable 3 the necessary RIA services to support the specified use cases shall be prototyped. Deliverable 1 the Scheduler shall be prototyped. Deliverable 2 the Twitter Search Agent shall be prototyped. Deliverable 3 the Search Session and Result shall be prototyped. Deliverable 4 the Scheduler shall be prototyped. Deliverable 5 the Twitter Search Agent shall be prototyped. 4/30/2010 5/31/ Inputs shall be validated and throttling shall be implemented. A test migration to the Azure platform will be carried out, and the first integration testing will be performed. 4. Rules shall be able to support Keywords, Date/Time, Locations, and Authors Filter. Integration Test 2 shall be performed. 5. User shall be able to search Digg in addition to Twitter. Integration Test 3 shall be performed. Deliverable 6 the Search Session and Results shall be prototyped. Deliverable 1 throttling shall be working. Deliverable 2 a test report detailing the testing plan and test results of Integration Test 1. Deliverable 3 a report detailing the test migration process and the experience. Deliverable 1 a test report detailing the testing plan and test results of Integration Test 2. Deliverable 1 a test report detailing the testing plan and test results of Integration Test Stress testing. Deliverable 1 a test report detailing the testing plan and test results of the Stress Test. 7. Final documentation and reflection. Deliverable 1 an operations manual detailing all functional use cases or a recorded demo. 6/30/2010 7/31/2010 9/30/ /31/ /30/2010

15 Deliverable 2 each developer shall submit a reflection of the experience gained from the development process. Deliverable 3 code generated documentation of key parts of the system. 7 Closing Summary Cloud Reader is a personalized search agent application that runs on the Windows Azure platform. This project is being done to obtain experience with developing a traditional application and migrating it to the Windows Azure platform. The first integration test will be performed after the initial prototype of the application and initial migration to the Windows Azure platform. Subsequent development will increase the number of Filters and Search Agent types in preparation for more robust integration and stress testing. The goal from performing these tests will be to glean insight into the performance of the application within the Windows Azure environment. Important deliverables once the project completes include team member reflections on the development lifecycle, testing reports, and the working application. 8 Project Team Information Clients Shawn Hanson shanso@microsoft.com Matt Thalman Matt.Thalman@microsoft.com Faculty Advisor Dr. Suraj C. Kothari 3132 Coover Hall Iowa State University Ames, IA Phone: kothari@iastate.edu Team Members Ben Holland 3206 Lincolnway Unit 1 Phone: bholland@iastate.edu Eric Petrowiak [Insert Address Here] Phone: ericp1@iastate.edu Dan Leeaphon Dan.Leeaphon@microsoft.com Bao Nguyen 4733 Toronto St. Unit 110 Phone: l3a0@iastate.edu Barret Schloerke 1711 Woodhaven Cir. Ames, IA Phone: bigbear@iastate.edu

16 9 Appendices Figure 1

17 Figure 2 Figure 3

18 Figure 4

19 Figure 5