i5k_doc Documentation Release 1.0 Fish Lin June 27, 2016
Table of Contents 1 Pre-requeisites 3 1.1 Python modules............................................. 3 1.2 Service-side pre-requisites........................................ 3 2 Setup guide 5 2.1 Environmnet............................................... 5 2.1.1 Python 2.7.8........................................... 5 2.1.2 RabbitMQ............................................ 6 2.1.3 Celery.............................................. 6 2.1.4 Memcached........................................... 7 2.2 Python Modules............................................. 7 2.2.1 django.............................................. 7 2.2.2 django-pipeline......................................... 8 2.2.3 Django REST framework.................................... 8 2.2.4 Django Suit........................................... 8 2.2.5 filebrowsers........................................... 9 2.2.6 python-social-auth....................................... 9 2.3 Database................................................. 9 2.3.1 Others.............................................. 10 2.4 Run on Apache HTTP Server...................................... 10 2.5 Continuous integration.......................................... 10 2.5.1 Jenkins............................................. 10 3 Workplace Apps 11 3.1 Blast................................................... 11 3.1.1 Install & Configuration..................................... 11 3.1.2 BLAST DB Configuration................................... 11 3.2 Hmmer.................................................. 12 3.2.1 Install & Configuration..................................... 12 3.2.2 HMMER DB Configuration.................................. 12 3.2.3 HMMER Query Histroy.................................... 13 3.3 Clustal.................................................. 13 3.3.1 Install & Configuration..................................... 13 3.3.2 Clustal Query Histroy...................................... 13 3.4 Dashboard................................................ 14 3.5 Data.................................................... 14 3.6 Proxy................................................... 14 3.7 Drupal_SSO............................................... 14 i
3.8 WebApollo SSO............................................. 14 4 WebApollo Single Sign On 15 4.1 What is WebApollo SSO?........................................ 15 4.2 Framework Overview.......................................... 15 4.3 Configuration............................................... 16 4.4 Register WebApollo........................................... 16 4.5 Utilities.................................................. 17 4.5.1 Utilities only for Admin.................................... 17 4.5.2 General Utilities......................................... 17 5 About i5k Workplace 19 5.1 About i5k................................................. 19 5.2 Contact.................................................. 19 6 Indices and tables 21 ii
This is my introduction to this project Requirements My project depend on xxx Contents: Table of Contents 1
2 Table of Contents
CHAPTER 1 Pre-requeisites 1.1 Python modules Descibed in requirements.txt Django==1.8.12 Markdown==2.6.6 celery==3.1.23 cssmin==0.2.0 django-pipeline==1.6.8 django-simple-captcha==0.4.5 djangorestframework==2.3.4 django-rest-swagger==0.3.5 docutils==0.12 jsmin==2.0.11 pillow==2.2.2 psycopg2==2.6 pycrypto==2.6.1 python-memcached==1.57 python-social-auth==0.2.16 requests-oauthlib==0.6.1 wsgiref django-suit==0.2.18 1.2 Service-side pre-requisites RabbitMQ mod_wsgi PostgreSQL 3
4 Chapter 1. Pre-requeisites
CHAPTER 2 Setup guide This setup guide is tested in Centos 7.0/6.5 and django 1.8.12 2.1 Environmnet 2.1.1 Python 2.7.8 Install necessary packages: yum groupinstall "Development tools" yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel yum install readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel python-devel Install python 2.7.8 from source: wget http://www.python.org/ftp/python/2.7.8/python-2.7.8.tar.xz xz -d Python-2.7.8.tar.xz tar -xvf Python-2.7.8.tar # Enter the directory: cd Python-2.7.8 # Run the configure:./configure --prefix=/usr/local # compile and install it: make make altinstall # Checking Python version: [root@i5k ~]# python2.7 -V Python 2.7.8 export PATH="/usr/local/bin:$PATH" Intall pip and virtualenv: wget https://bootstrap.pypa.io/ez_setup.py python2.7 ez_setup.py wget https://bootstrap.pypa.io/get-pip.py python2.7 get-pip.py pip2.7 install virtualenv 5
Build a separated virtualenv: # create a virtual environment called py2.7. # Activate it by source py2.7/bin/activat # Make dir '/path/to/i5k/virtualenv' and switch in mkdir /path/to/i5k/virtualenv cm /path/to/i5k/virtualenv /usr/bin/virtualenv py2.7 source py2.7/bin/activate 2.1.2 RabbitMQ Install RabbitMQ Server: ## RHEL/CentOS 7 64-Bit ## wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm rpm -ivh epel-release-7-5.noarch.rpm #Install Erlang: yum install erlang #Install RabbitMQ server: yum install rabbitmq-server #To start the daemon by default when system boots, as an administrator run: #chkconfig rabbitmq-server on systemctl enable/disable rabbitmq-server #To start/stop server: #/sbin/service rabbitmq-server start/stop/restart/status systemctl start/stop/restart/status rabbitmq-server 2.1.3 Celery Tested on Celery 3.1.23 pip install celery==3.1.23 pip install django==1.8.12 Copy init celery script and celerybeat script to /etc/init.d/ Copy two configuration files (/path/to/i5k/celeryd.sysconfig and celerybeat.sysconfig) to /etc/default/, modify CELERYD_CHDIR, CELERYD_MULTI, CELERYBEAT_CHDIR and CELERY_BIN with your project path. Modify code in celeryd init script as following. # Change code in celeryd init script if [[ `dirname $0` == /etc/rc*.d ]]; then SCRIPT_FILE=$(readlink "$0") else SCRIPT_FILE="$0" fi # To 6 Chapter 2. Setup guide
if [[ -L "$0" ]]; then SCRIPT_FILE=$(readlink "$0") else SCRIPT_FILE="$0" fi # cp celeryd init script to /etc/init.d/ # cp celerybeat init script to /etc/init.d/ cp /path/to/i5k/celeryd.sysconfig /etc/default/celeryd cp /path/to/i5k/celerybeat.sysconfig /etc/default/celerybeat # set as daemon chkconfig celeryd on chkconfig celerydbeat on 2.1.4 Memcached Queue status can be provided by installing and activating memcached while query is submitted. In settings.py, change USE_CACHE=True. reference Install and activated memcached: yum install memcached # chkconfig memcached on # service memcached restart systemctl enable memcached ststemctl start memcached Configuration in setting.py: CACHES = { 'default': { 'BACKEND': 'django.core.cache.backends.memcached.memcachedcache', 'LOCATION': '127.0.0.1:11211', 'TIMEOUT': None, # never expire } } 2.2 Python Modules 2.2.1 django Install django 1.8.12 pip intall django==1.8.12 Authentication backend All authenticaion and authorization stuff such as login, password, permissions and users are handled by Django s default authentication backend. We use or extend Django s classes to build user-related functions in app/views.py, and put the url mappings in i5k/urls.py and HTML files in app/templates/app/*.html. The data tables Auth -> Users and Auth -> Groups are built-in for storing user data. 2.2. Python Modules 7
django-axe django-axes is a very simple way for you to keep track of failed login attempts, both for the Django admin and for the rest of your site. User login fail log could be viewed in Admin page (Axe -> Lockout status). Change Failed logins field or delete record for unlock this account. AXES_LOGIN_FAILURE_LIMIT, the number of login attempts allowed before a record is created for the failed logins. Default 3. 2.2.2 django-pipeline Install django-pipeline pip install django-pipeline==1.6.8 Configuration in setting.py: INSTALLED_APPS = ( 'pipeline', ) STATICFILES_STORAGE = 'pipeline.storage.pipelinecachedstorage' STATICFILES_FINDERS = ( 'django.contrib.staticfiles.finders.filesystemfinder', 'django.contrib.staticfiles.finders.appdirectoriesfinder', 'pipeline.finders.pipelinefinder', 'pipeline.finders.cachedfilefinder', ) 2.2.3 Django REST framework Install rest framework (rest framework is still ongoing) pip install djangorestframework==2.3.4 pip install django-rest-swagger==0.3.5 2.2.4 Django Suit Install django suit pip install django-suit==0.2.18 Configuration in setting.py: INSTALLED_APPS = ( 'suit', 'django.contrib.admin', ) TEMPLATES = [ { 'OPTIONS': { 8 Chapter 2. Setup guide
] } }, 'context_processors': [ 'django.template.context_processors.request', ], 2.2.5 filebrowsers 2.2.6 python-social-auth Install social-auth relatives pip install requests-oauthlib==0.6.1 pip install python-social-auth==0.2.16 Supported by python-social-auth package. You will need a google account and a facebook account to setup your app and get the keys and secrets for the app. Then fill the following section in settings_prod.py. Refer to this instruction for detailed settings. # social login settings SOCIAL_AUTH_GOOGLE_OAUTH2_KEY = '' SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET = '' SOCIAL_AUTH_FACEBOOK_KEY = '' SOCIAL_AUTH_FACEBOOK_SECRET = '' Tables and data under Social Auth are automatically generated. Social Auth -> User social auths stores mappings between users OAuth data and their Django user accounts. Some notes about current python-social-auth settings: Social and local accounts are corresponded by email address. Different social accounts with the same email would be mapped to the same local user in Django. User can register a local account through our registration form, or the system would create a new one for the user who first logs in with his/her social account. Only the former can update their account information (the later can only update their institutions), change their password and request for password reset. 2.3 Database Using PostgreSQL as Database Backend: # install PostgresSQL postgres=# create user django; postgres=# create database django; postgres=# grant all on database django to django; # connect to database django postgres=# \c django # create extension hstore postgres=# create extension hstore; # config in pg_hba.conf pip install psycopg2==2.6 2.3. Database 9
Install PostgreSQL ( postgresql reference ) Create the user and database for this application Allowing connection from the database user by setting your /var/lib/pgsql/9.x/data/pg_hba.conf Install pip package psycopg2. Path of pg_config binary may need to be exported. pip install psycopg2==2.6 export PATH=/usr/pgsql-9.x/bin:"$PATH" Migration db schema to PostgreSQL python manage.py migrate 2.3.1 Others Install necessary modules pip install -r /path/to/i5k/requirements.txt 2.4 Run on Apache HTTP Server Install Apache HTTP Server and development tools ( install reference ) yum install httpd-devel # set as daemon systemctl enable httpd wget https://github.com/grahamdumpleton/mod_wsgi/archive/4.4.23.tar.gz tar -zxf 4.4.23.tar.gz cd mod_wsgi-4.4.23/./configure --with-apxs=/usr/sbin/apxs make make install Use Django with Apache and mod_wsgi ( configuration reference ) LoadModule wsgi_module modules/mod_wsgi.so 2.5 Continuous integration 2.5.1 Jenkins 10 Chapter 2. Setup guide
CHAPTER 3 Workplace Apps 3.1 Blast Introduction I5K BLAST Tutorial is on https://i5k.nal.usda.gov/content/blast-tutorial 3.1.1 Install & Configuration Install BLAST and append Blast_bin directory in environment variable PATH. 3.1.2 BLAST DB Configuration There are five tables for creating BLAST DB and browsing in I5K-blast. Add Organism: Display name should be scientific name. Short name are used by system as a abbreviation. Descriptions and NCBI taxa ID are automatically filled. Add Sequence types: Used to classify BLAST DBs in distinct catagories. 11
Provide two kinds of molecule type for choosing, Nucleotide/Peptide. Add Sequence Add BLAST DB Choose Organsim Choose Type (Sequence type) Type location of fasta file in FASTA file path Type Title name. (showed in HMMER page) Type Descriptions. Check is shown, if not check, this database would show in HMMER page. Save Add JBrowse settings 3.2 Hmmer HMMER is used for searching sequence databases for homologs of protein sequences, and for making protein sequence alignments. It implements methods using probabilistic models called profile hidden Markov models (profile HMMs). I5K HMMER Tutorial is on https://i5k.nal.usda.gov/webapp/hmmer/manual. 3.2.1 Install & Configuration Install HMMER and append HMMER_bin directory in evironment varialbe PATH. 3.2.2 HMMER DB Configuration Like Blast, HMMER databases must be configured then they could be searched. 12 Chapter 3. Workplace Apps
Go django admin page and click Hmmer on left-menubar. You need to create HMMER db instance (Hmmer dbs) for each fasta file. Choose Organsim Type location of peptide fasta file in FASTA file path Type Title name. (showed in HMMER page) Type Descriptions. Check is shown, if not check, this database would show in HMMER page. Save 3.2.3 HMMER Query Histroy HMMER query histories are stored in table HMMER results. Users could review them on dashboard. All query results (files on disk) will be removed if it s expired. (default: after seven days) Query results locate in directory $MEDIA_ROOT/hmmer/task/. 3.3 Clustal ClustalW is the oldest of the currently most widely used programs for multiple sequence alignment. Clustal Omega is the latest version of CLUSTAL series. ClustalO is faster and more accurate because of new HMM alignment engine. I5K CLUSTAL Tutorial is on https://i5k.nal.usda.gov/webapp/clustal/manual. 3.3.1 Install & Configuration Install Clustalw and Clustal Omega. Then append both bin directory in evironment varialbe PATH. 3.3.2 Clustal Query Histroy Clustal query histories are stored in table Clustal results. Users could review them on dashboard. All query results (files on disk) will be removed if it s expired. (default: after seven days) Query results locate in directory $MEDIA_ROOT/clustal/task/. 3.3. Clustal 13
3.4 Dashboard Personal query history. 3.5 Data Rest framework. Not finished 3.6 Proxy For providing indirect access to some resources without https. Currently it is used by Web Apollo instances for looking up GO Terms. 3.7 Drupal_SSO Coonection to Drupal summit data function. DRUPAL_URL = 'https://gmod-dev.nal.usda.gov' # cookie can be seen in same domain DRUPAL_COOKIE_DOMAIN=".nal.usda.gov" 3.8 WebApollo SSO Complete introduction locate in Section 4. 14 Chapter 3. Workplace Apps
CHAPTER 4 WebApollo Single Sign On 4.1 What is WebApollo SSO? The basic idea in SSO is to provide handy user interface and make WebApollo user more like a community. In order to accomplish those ideas, we try to transfer management jobs from WebApollo to SSO. SSO gives the coordinators more authority to manage their members who can annotating and grant the priviledges on their own. In SSO, we seperate users into three different roles. First, the ADMIN who actually owns admin priviledge in WebApollo, can manage users/groups/eroll event. Second, the COORDINATOR who belong to group GROUP_(Organism_short_name(OSN))_ADMIN, can manage membership in specific (Organism). Last, the remaining users are in USER. They can make request to join (or leave) different organism team. Once be recuited in, user will pertain to group GROUP_(OSN)_USER. SSO make a virtual role COORDINATOR by exploiting a conventional group name GROUP_(OSN)_ADMIN and the user in the team would be in group GROUP_(OSN)_USER. Role\ WebApollo Single Sign On (SSO) ADMIN Global Admin Global Admin COORDINATOR Admin permission in GROUP_()_ADMIN USER RWE permission in GROUP_()_USER with RWE permission Note: Mapping between full organism name and short organism name are stored in django-blast app. Full organism name is the real name in WebApollo and short name is a abbreviation used in django-blast app. 4.2 Framework Overview SSO was implemented in Django and JQuery. Conceptually, SSO is a proxy service for delegating user request to appropriate WebApollo service. The main advantage here is that SSO could provides more social utilities for the I5K community. 15
Database Schema (UserMapping) Apollo_user_id Apollo_user_name Apollo_user_pwd django_user last_date 1 Chris (AES encrpted pwd) Christopher 2 Monica (AES encrpted pwd) Monica 3 Mei (AES encrpted pwd) NULL SSO records the mapping between Apollo_user and django_user in table UserMapping. Apollo_user_id and django_user are unique attribute and this makes mapping a one to one relationship. (apollo_user_name could be changed and is not unique) In above table, record 1 and 2 tell a formal relationship but record 3 describes an Apollo user doesn t belong to any django user. User can claim it by re-register process. (mentioned below) 4.3 Configuration SSO uses a pre-assigned admin Apollo account to communiate with Apollo server. The account must be create on apollo server first. Two URLs address of i5k server and apollo server are used to identify each others locations. In order to secure user password, SSO encrpt it before saving password into database. WebApollo SSO configuration in django setting.py: # WebApollo SSO robot account ROBOT_ID='R2D2' ROBOT_PWD='demo' #URL of i5k workspace and webapollo I5K_URL='http://i5k.nal.gov' APOLLO_URL='http://i5k.apollo.nal.gov/apollo' # cookie can be seen in Apollo-prod and Gmod-prod APOLLO_COOKIE_DOMAIN=".nal.usda.gov" #Encypt webapollo user password in SSO database. #AES key must be either 16, 24, or 32 bytes long. SSO_CIPHER='1234567890123456' 4.4 Register WebApollo There are three ways to make connection between i5k account to apollo account. 16 Chapter 4. WebApollo Single Sign On
When registering an new i5k account, SSO also create an apollo account(same ID). When entering SSO, if SSO doesn t have mapping record of user, it asks user to create a new apollo account or register his account info into SSO. When entering SSO, if SSO has mapping record of user but login failed, it asks user to re-enter his password into SSO. 4.5 Utilities There are six individual tab pages, three of them are general and others are specific for Admin user. 4.5.1 Utilities only for Admin Tab\ User(Admin) Group(Admin) PReq(Admin) Function Descriptions View/Create/Delete/Update/Disconnect Apollo User View/Create/Delete Apollo Group View Pending request 4.5.2 General Utilities Tab\ My Organism My Request My Info Function Descriptions Manage organism which you joined in / Go WebApollo Make request to join/leave a organism community User basic information 4.5. Utilities 17
18 Chapter 4. WebApollo Single Sign On
CHAPTER 5 About i5k Workplace 5.1 About i5k we are i5k group 5.2 Contact xxxx 19
20 Chapter 5. About i5k Workplace
CHAPTER 6 Indices and tables genindex modindex search 21