Phase Documentation. Release 0.1. ChangeToMyName

Transcription

1 Phase Documentation Release 0.1 ChangeToMyName May 18, 2015

2

3 Contents 1 Getting started Introduction Installation Available fabric commands Basic usage 3 3 Customizing document models Document model definition Required fields and methods Document unique identifier Document list columns Search and filter form Phase deployment Hosting Phase on a dedicated server Server installation NodeJS installation Database creation Python configuration Elasticsearch configuration Phase installation Web server configuration Running the application Cronjobs Reindex all Crontab Transmittals upload Directory definition Server configuration Colophon 19 i

4 ii

5 CHAPTER 1 Getting started 1.1 Introduction Phase is a document management system specifically designed for the needs of engineering and construction projects to manage the documentation of oil & gas, water treatment, nuclear, solar and wind facilities. Phase offers the following characteristics: Management of document and data lists containing thousands of items Management of multiple metadata related to engineering, review, schedule, etc. Spreadsheet like filtering/search capabilities Document and data versioning Management of relationships between documents and data Phase is intended to be used on projects where: Thousands of documents are generated Documents have to be produced, exchanged, reviewed, revised and used all along the project phases by multiple parties (owner/operator, contractors, vendors, partners, authorities, etc.) 1.2 Installation Steps to initialize the project on a local machine: $ git clone repository $ virtualenv venv $ source venv/bin/activate $ pip install -r repository/requirements/local.txt $ npm -g install yuglify $ cd repository $ python phase/manage.py syncdb --noinput --settings=core.settings.local $ fab runserver 1.3 Available fabric commands Check the list of available commands directly in your shell: 1

6 $ fab -l Available commands: check deploy docs errors log runserver test Checks that everything is fine, useful before deploying. Deploys the project against staging. Generates sphinx documentation for the project. Displays error.log file from staging. Displays access.log file from staging. Runs the local Django server. Launches tests for the whole project. 2 Chapter 1. Getting started

7 CHAPTER 2 Basic usage TODO : Define phase usage Category creation Category templates 3

8 4 Chapter 2. Basic usage

9 CHAPTER 3 Customizing document models Phase comes with predefined document models. However, it is designed so you can create your own. All you need to do is to create a new application with a name ending by _documents : mkdir myproject_app cd myproject_app django-admin.py startapp myproject_documents You need to make sure that this application is accessible in the PYTHONPATH. If you use virtualenvwrapper, you can use add2virtualenv: add2virtualenv myproject_app Once this is done, add your application to the INSTALLED_APPS list and run syncdb. 3.1 Document model definition Every document model is made of two classes: a base metadata class and a revision class. The base class must inherit of documents.models.metadata and the revision class must inherit of documents.models.metadatarevision. Here s a basic sample: class DemoMetadata(Metadata): # This field with this exact name is required latest_revision = models.foreignkey( 'DemoMetadataRevision', verbose_name=_('latest revision'), null=true) title = models.charfield( _('Title'), max_length=50) leader = models.foreignkey( User, verbose_name=_('leader'), related_name='leading_demo_metadata', null=true, blank=true) class Meta: ordering = ('title',) # This is a custom class, required to configure the document model 5

10 class PhaseConfig: # Here are the fields that fill appear in the filter form filter_fields = ( 'leader', ) # Those fields will be searchable in the filter form searchable_fields = ( 'document_key', 'title' ) # Column definition column_fields = ( ('Document Number', 'document_key'), ('Title', 'title'), ('Rev.', 'current_revision'), ('Rev. Date', 'current_revision_date'), ('Status', 'status'), ) def natural_key(self): return (self.document_key,) def generate_document_key(self): return slugify(self.title) class DemoMetadataRevision(MetadataRevision): STATUSES = ( ('opened', 'Opened'), ('closed', 'Closed'), ) native_file = RevisionFileField( _('Native File'), null=true, blank=true) pdf_file = RevisionFileField( _('PDF File'), null=true, blank=true) status = models.charfield( _('Status'), max_length=20, choices=statuses, null=true, blank=true) Some fields and methods must be defined on the base class. 3.2 Required fields and methods On the metadata base class, you must define a latest_revision field as a foreign key to the corresponding metadata class. Inside this class, you also must define a PhaseConfig class the same way you would define a Meta class. This is used to configure how your document model integrates itself into Phase. To have the full list of methods that you must implement, take a look in documents/models.py and check all methods that throw a NotImplementedError. 6 Chapter 3. Customizing document models

11 3.3 Document unique identifier Every document in Phase have a unique identifier, stored in the document_key field. However, every document type must define how this field is generated. This must be done in the generate_document_key method. Here is a example : def generate_document_key(self): return slugify( u"{contract_number}-{originator}-{unit}-{discipline}-" u"{document_type}-{sequential_number}".format( contract_number=self.contract_number, originator=self.originator, unit=self.unit, discipline=self.discipline, document_type=self.document_type, sequential_number=self.sequential_number )).upper() The fields that you will use to build unique identifiers should also be listed in a unique_together entry in the Meta subclass. 3.4 Document list columns In PhaseConfig, the column_fields is used to define which fields will be displayed inside columns. column_fields = ( ('Document Number', 'document_key', 'document_key'), ('Title', 'title', 'title'), ('Rev.', 'current_revision', 'latest_revision.revision'), ('Rev. Date', 'current_revision_date', 'latest_revision.revision_date'), ('Status', 'status', 'latest_revision.status'), ) Each entry is composed of three elements: 1. The name that will be displayed in the column header. 2. The class that will be given to the column. 3. The accessor to get the column value. You can use a field name or a property. 3.5 Search and filter form In the document list, a document filter form is displayed to search and filter documents. Which field will be used is also defined in PhaseConfig. # Here are the fields that fill appear in the filter form filter_fields = ('leader',) # Those fields will be searchable in the filter form # You can use fields from the base document or the revision searchable_fields = ('document_key', 'title') 3.3. Document unique identifier 7

12 8 Chapter 3. Customizing document models

13 CHAPTER 4 Phase deployment Phase is designed to be a lightweight alternative to traditional bloated and slow DMS. Hence a Phase instance can be run on a single virtual machine. A single dedicated server can host several environments (pre-production, production). 4.1 Hosting Phase on a dedicated server If you choose to manage your own dedicated server, you can use OpenVZ containers to contain different Phase installations. Follow the OpenVZ installation instructions. 4.2 Server installation If you created an OpenVZ container from the debian wheezy template, you need to install the following packages: apt-get update apt-get upgrade apt-get purge apache2 apt-get install build-essential libpq-dev python-dev apt-get install postgresql postgresql-contrib nginx git supervisor rabbitmq-server 4.3 NodeJS installation Some tools used in Phase require a node.js installation. Get the latest version url on the Node.js site. Let s install it: cd /opt/ wget tar -zvxf node-v tar.gz cd node-v /configure make make install 9

14 4.4 Database creation su - postgres createuser -P Enter name of role to add: phase Enter password for new role: phase Enter it again: phase Shall the new role be a superuser? (y/n) n Shall the new role be allowed to create databases? (y/n) n Shall the new role be allowed to create more new roles? (y/n) n createdb --owner phase phase 4.5 Python configuration Install pip and virtualenv: cd wget python get-pip.py pip install virtualenv virtualenvwrapper Create user: adduser phase --disabled-password su - phase Add those lines in the ~/.profile file: export WORKON_HOME=~/.virtualenvs mkdir -p $WORKON_HOME source `which virtualenvwrapper.sh` Then: source ~.profile 4.6 Elasticsearch configuration Phase uses Elasticsearch to index documents and provides search features. The default Elasticsearch installation is enough, but remember that ES listens on by default, which can be inconveniant. To limit ES connections to localhost, one can update the config file /etc/elasticsearch/elasticsearch.yml as is:... network.host: You also need to make sure that your virtual machine has enough memory available. 10 Chapter 4. Phase deployment

15 4.7 Phase installation As root: npm install -g cssmin uglify-js As phase user: cd mkvirtualenv phase git clone cd phase/src pip install -r../requirements/production.txt export DJANGO_SETTINGS_MODULE=core.settings.production python manage.py collectstatic python manage.py syncdb 4.8 Web server configuration If you don t host any other site on the same server, you can replace nginx s default virtual host in /etc/nginx/sitesavailable/default: server { listen 80 default_server; return 444; } Create the Phase configuration file in /etc/nginx/sites-available/phase. Here is a working sample. upstream phase { server localhost:8000; } server { server_name app.url.com; access_log /var/log/nginx/phase.access.log; error_log /var/log/nginx/phase.error.log; location /static/ { alias /home/phase/phase/public/static/; } location /media/ { alias /home/phase/phase/public/media/; } # Phase use x-sendfile / x-accel to serve protected file # directly using the server. location /xaccel/ { internal; alias /home/phase/phase/private/; } location / { proxy_pass proxy_redirect off; 4.7. Phase installation 11

16 } } proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; Then create a link to enable it: ln -s /etc/nginx/sites-available/phase /etc/nginx/sites-enabled/ Don t forget to restart nginx: /etc/init.d/nginx restart 4.9 Running the application Gunicorn is the recommanded WSGI HTTP server to run Phase. Supervisor will be used to monitor it. Create the /etc/supervisor/conf.d/phase.conf config file. here is a working sample. [program:phase] environment=django_settings_module='core.settings.production' directory=/home/phase/phase/src command=/home/phase/.virtualenvs/phase/bin/python manage.py run_gunicorn user=phase autostart=true autorestart=true stdout_logfile=/var/log/supervisor/phase.log redirect_stderr=true Phase uses celery as a task queue. Here is the corresponding supervisor file. [program:celery] environment=django_settings_module='core.settings.production' directory=/home/phase/phase/src/ command=/home/phase/.virtualenvs/phase/bin/celery -A core.celery worker -l info user=phase numprocs=1 stdout_logfile=/var/log/celery_stdout.log stderr_logfile=/var/log/celery_stderr.log autostart=true autorestart=true startsecs=10 Run this thing with: supervisorctl reread supervisorctl reload 12 Chapter 4. Phase deployment

17 CHAPTER 5 Cronjobs 5.1 Reindex all Documents must be reindexed every day so that elastic search can stay in synch with actual document data. There is a dedicated task for it: python manage.py reindex_all Warning: This task will completely delete the index and recreate it from scratch. 5.2 Crontab Setup a crontab to run scheduled tasks regularly. You must use your phase user to run the tasks. Here is a sample crontab file: PYTHON="/home/phase/.virtualenvs/phase/bin/python" DJANGO_PATH="/home/phase/phase/src/" LOGS_PATH="/home/phase/django_logs/" DJANGO_SETTINGS_MODULE="core.settings.production" # m h dom mon dow command * * * cd $DJANGO_PATH && $PYTHON manage.py reindex_all --noinput &>"$LOGS_PATH/reindex.log" 13

18 14 Chapter 5. Cronjobs

19 CHAPTER 6 Transmittals upload The transmittals upload feature allows a contractor to upload a bunch of documents into a Phase instance directly from a ftp upload. 6.1 Directory definition The directory must be named XXX dir content 6.2 Server configuration Here are the instructions to install and configure the ftp server to activate this feature. Note that Phase doesn t care how the files are transmitted to the server (ftp, ssh, nfs, etc.) so this section is for information only Ftp server installation and configuration We will use the proftpd server to handle ftp communication, and configure the server to only accept ftps (ftp over ssl) connexions. First, install the proftpd ftp server: aptitude install proftpd Choose the standalone start method. Create the ssl certificates for the TLS connection. openssl req -x509 -newkey rsa:1024 \ -keyout /etc/ssl/private/proftpd.key -out /etc/ssl/certs/proftpd.crt \ -nodes -days 365 chmod 0600 /etc/ssl/private/proftpd.key chmod 0640 /etc/ssl/private/proftpd.key Configure the server, using those examples files as starting points. /etc/proftpd/proftpd.conf : 15

20 # Includes DSO modules Include /etc/proftpd/modules.conf # Set off to disable IPv6 support which is annoying on IPv4 only boxes. UseIPv6 off RootLogin off # If set on you can experience a longer connection delay in many cases. IdentLookups off ServerName ServerType DeferWelcome MultilineRFC2228 DefaultServer ShowSymlinks on "Phase" standalone off on on TimeoutNoTransfer 600 TimeoutStalled 600 TimeoutIdle 1200 DisplayLogin DisplayChdir ListOptions welcome.msg.message true "-l" DenyFilter \*.*/ # Use this to jail all users in their homes DefaultRoot ~ # Users require a valid shell listed in /etc/shells to login. # Use this directive to release that constrain. RequireValidShell off # Port 21 is the standard FTP port. Port 21 # To prevent DoS attacks, set the maximum number of child processes # to 30. If you need to allow more than 30 concurrent connections # at once, simply increase this value. Note that this ONLY works # in standalone mode, in inetd mode you should use an inetd server # that allows you to limit maximum number of processes per service # (such as xinetd) MaxInstances 30 # Set the user and group that the server normally runs at. User proftpd Group nogroup # Umask 022 is a good standard umask to prevent new files and dirs # (second parm) from being group and world writable. Umask # Normally, we want files to be overwriteable. AllowOverwrite off 16 Chapter 6. Transmittals upload

21 # This is required to use both PAM-based authentication and local passwords # AuthOrder mod_auth_pam.c* mod_auth_unix.c TransferLog /var/log/proftpd/xferlog SystemLog /var/log/proftpd/proftpd.log # In order to keep log file dates consistent after chroot, use timezone info # from /etc/localtime. If this is not set, and proftpd is configured to # chroot (e.g. DefaultRoot or <Anonymous>), it will use the non-daylight # savings timezone regardless of whether DST is in effect. SetEnv TZ :/etc/localtime DelayEngine on # This is used for FTPS connections Include /etc/proftpd/tls.conf # List of authorized users Include /etc/proftpd/users.conf # Prevent files and directories rename / deletion <Limit DELE> DenyAll </Limit> <Limit RNFR> DenyAll </Limit> <Limit RNTO> DenyAll </Limit> /etc/proftpd/tls.conf : TLSEngine TLSRequired TLSProtocol TLSVerifyClient TLSRSACertificateFile TLSRSACertificateKeyFile TLSLog on on SSLv23 off /etc/ssl/certs/proftpd.crt /etc/ssl/private/proftpd.key /var/log/proftpd/tls.log /etc/proftpd/users.conf : <Limit LOGIN> AllowUser test_ctr DenyALL </Limit> User creation Let s create a unix user test_ctr for the contractor, and configure the directory permissions Server configuration 17

22 adduser test_ctr --disabled-password --ingroup=phase --shell=/bin/false chmod g+rwx /home/test_ctr echo "umask 002" >> /home/test_ctr/.profile Note that for safety reasons, the list authorized users are explicitely declared in the /etc/proftpd/users.conf file. 18 Chapter 6. Transmittals upload

23 CHAPTER 7 Colophon This documentation is generated by sphinx, please edit docs/index.rst to add more content and use the fab docs command to compile it. Django: Bootstrap: Two Scoops of Django template: Sphinx: Datepicker for Bootstrap: File upload for Bootstrap: jquery UI MultiSelect Widget: yuglify: 19