CGHub Web-based Metadata GUI Statement of Work

Transcription

1 CGHub Web-based Metadata GUI Statement of Work Mark Diekhans Version 1 April 23, Goals CGHub stores metadata and data associated from NCI cancer projects. The goal of this project is to develop a web-based metadata browsing and search application. It will be used for locating and selecting data files for download. This application is referred to as the CGHub Browser. The CGHub Browser does not initiate or control the download. It produces a manifest file that will be passed to another program for download. The metadata is publicly accessible, which requires no authentication mechanism to view the metadata. Easy-of-use for basic users, who do not understand the details of the how the metadata is organized, is a priority. The metadata is hierarchical, conforming to the SRA 1.3 XML schema. Data files may be referenced at various levels in the metadata tree. Nodes may be missing from the tree due to incorrect or inconsistent metadata. A project is underway to correct these problems; however, the CGHub Browser should be able to function in the absence of required metadata nodes or terms. The browser is based entirely by the SRA metadata and other metadata available through the CGHub web services API. A secondary goal is to produce a set of Python class providing object-oriented API to the CGHub web services API (WS-API). This Python API is to be used in the development of the CGHub Browser and to support development of other applications. 2 Requirements This section outlines the initial requirements for the two major components of the project. Requirements may evolve as the project progresses. All code will be developed and made available under the new BSD license. Care should be taken to not to bring in dependencies that conflict with this license. 2.1 CGHub Browser 1. The CGHub Browser will be implemented using the public WS-API, allowing the application to run outside of the CGHub firewall. Enhancements will be proposed to the WS-API as needed to support this project. 2. UCSC has done a prototype of the CGHub Browser, as shown in Figures 1 through 6. It was built using the Django admin facility on top of a database import of the metadata. It will be used as a starting point for the GUI design. The developers will work with the CGHub staff to refine this design. We will give access to the running prototype and the code once we have transitioned it to a new test system. 1

2 3. All access to the browser is anonymous. A cookie-based mechanism can be used to save state between browsing sessions, although long-term persistence is not required. 4. The CGHub Browser will use a shopping cart type of paradigm, where users browse and select BAMs or other data files for download. Selection adds the data to the cart. At any point, the user may review and modify the contents of their cart. The cart should provide a summary of the number and total size of data to be downloaded. 5. A user can export the cart to a manifest file the can be consumed by the GeneTorrent client program. 6. A user can also download the XML metadata from the cart. 7. The browser will provide search, filter, and sort functions. The initial set will include those in the prototype. It should be easy to add new searches by writing simple server-side Python modification. 8. The ability to launch the browser with a configured set of filters and sorting given only a URL should be supported. This will allow us to provide a link that shows what data has been recently added. 9. UCSC will define a basic look and feel that is consistent with the UCSC Cancer Genome browser All web pages will be in valid HTML All JavaScript code can assume at least version 1.8 features and avoid non-standard constructs that cause portability issues. Use of the JQuery package can be considered if desired by the developer. 12. All documentation of the CGHub Browser will be done as online, context-sensitive help. This will include an introductory document that outlines the use and work-flow of the application.. A built-in tutorial using the methodology of the UCSC Cancer Browser will also be included. The help facility should be developed along with the code. No documentation outside of the help facility should be required or written. 13. The project will be implemented in Python 2.7 using the Django web framework. Consideration should be take on making the application easy to port to Python The CGHub Python API developed, as part of this will be used to implement all access to the WS- API. 15. We will discuss the cost/benefit trade-offs of developing automated web-based functional tests of the application, most likely using Selenium. 16. Coding paradigm should favor more, small components, especially very small functions and loops, over larger components. Objects should naturally represent concepts. 2.2 CGHub Python API The Python API to the CGHub WS-API (CGHub Python API) will be designed and implemented covering the entire WS-API, not just the portions required by the CGHub Browser. It will also include a Python representation of the subset of the INSDC SRA schema used by CGHub. While the API should be done in a method that is natural to the Python programmer, it will serve as a guide for implementations for other languages, so general object oriented concepts should be favored over those highly specific to Python. The web services API is documented in the v1 6.pdf The INSDC SRA XML 1.3 metadata schema is available from doc/sra_1-3/ and is documented at and 2

3 1. The CGHub Python API will provide an object abstraction for requests to CGHub. It will have specific methods to issue each support request to CGHub. Results should either be an object parsed from the resulting XML or an exception on error. 2. Result objects should represent the XML as a hierarchical object, with fields, reflecting the attributes of the XML, not just as dictionary. 3. CGHub only manages INSDC SRA 1.3 experiment, run, and analysis objects and class should be defined to encapsulate these classes. These should be usable directly on XML files conforming to the schema as well as responses from the WS-API 4. We suggest looking at or other tools to generate the SRA classes from the XSD schema. 5. The API should be documented using the Sphinx package 6. The API should be implemented using Test-driven development practices and include building a complete unit test suite. 7. A performance test suite should be developed to assist in tuning the system. 8. The code for the CGHub Python API will be independent of Django and will be packaged for client install for building applications 9. Coding paradigm should favor more, small components, especially very small functions and loops, over larger components. Objects should naturally represent a concept. 3 Implementation Both parts of this project will follow an incremental, iterative implementation and review cycle, with frequent feedback and discussion with the CGHub staff. We view this as a collaborative effort, with iterative engineering especially important for developing a good user interface. Once the CGHub Browser has reached a sufficient level of functionality, the CGHub staff will arrange for alpha testing and feedback from a diverse set of actual users. While the actually implementation of the CGHub Browser is dependent upon completion of the CGHub Python API, design work can proceed in parallel if resources permit. CGHub estimated efforts, in engineer-days, are included below. Elapsed time will depend upon resource availability. We ask the developer to assign schedule estimates to their portions of the development implementation outlined below. While the below implementation phases are presented in a waterfall type of outline for simplicity, it s assumes that any previous stage may be revisited as required. 3.1 CGHub Python API Development Design phase. Estimate three cycles. ¾ Developer implements skeleton packages and classes and generates documentation. ¾ CGHub reviews documentation and makes recommendations. [2 days] Implementation phase. ¾ Basic functionality. Estimate three cycles. 3

4 Developer develops unit tests and code for one simple query and response CGHub reviews code to ensure that it matches CGHub s expectations. [2 days] ¾ Complete functionality. Estimate two cycles. Developer develops unit tests and code for remainder of functionality CGHub reviews code and test suite. [3 days] Performance phase. ¾ Development performance test suite ¾ Produce report on WS-API performance and pass to CGHub staff for possible tuning of the WS-API. 3.2 CGHub Browser Development Phasing Initial design phase. Estimate two cycles. ¾ CGHub providers material for look and feel. [1 day] ¾ Developer implements mock-up of main browser. This only needs to be enough functionality to evaluate the basic lock and feel. Design and implement phases. Estimate three-to-six cycles. ¾ Developer implements some subset of remaining functionality to allow for review. This should include help. Tutorial can be implemented in later cycles once flow of application is complete. ¾ CGHub staff experiments with implementation and gives feedback. [2 days] ¾ Alpha test phase ¾ CGHub arranges for testing by selected users. Feedback, from users, may result in more design and implementation phases. Beta test phase ¾ CGHub deploys for all users and solicits feedback. 4

5 Figure 1: Default Listing: Default view will be either latest uploaded bams (shown) or the last list the user displayed. 5

6 Figure 2: Default Listing (continued): Note the pagination in the lower left corner, and the list of diseases as a filter on the right side of the screen. 6

7 Figure 3: Search: User can search for a part of a barcode or filename, other attributes, or arbitrary text in the XML metadata. Note the list is reduced to 350 out of

8 Figure 4: Search and Filter: We now use two filter criteria to reduce the list further: (1) We chose to only list BAMs from WashU (Washington University) AND (2) We only want ones (BAMs) uploaded this year now we have the list down to 18/

9 Figure 5: Selection: Select BAM pairs to be downloaded. 9

10 Figure 6: Export Selection: This is for download metadata of the BAM files selected and for getting a GeneTorrent download file (both in the compressed archive). 10