itixi Ubuntu Server Deployment How-To/Information

Transcription

1 itixi Ubuntu Server Deployment How-To/Information Reto Schelbert 20. August 2014

2 1 Index 1 Index Virtual Server Information User/Root MySQL User Ubuntu Server Installation Mysql 5.6 installation Update and Upgrade Install Webmin (easier to configure server instances) Install PHP Modules curl xdebug Change PHP configuration Install wkhtmltopdf Install phpdoc (optional) Apache Config Configuration HTTP/Default Directory SSL Konfigurieren Konfiguration HTTPS/SSL Apache Webmin Symfony/sfitixi Install OSRM Install Build OSRM OSRM Extract and Prepare OSRM API OSRM Website Virtual Box NAT-Rules SSH HTTP Website HTTPS Website OSRM API Webmin Server Scripts tixi_update_and_rebuild.sh tixi_rebuild_server.sh osrm_update.sh osrm_start.sh osrm_stop.sh MySQL Backup Restore August 2014 Seite 1 von 48

3 17 New Client/Mandant New MySQL database Symfony/sfitixi instance Set parameters Set TIXI parameters run_build or deploy Apache directory VirtualBox Import August 2014 Seite 2 von 48

4 2 Virtual Server Information Download (as per ) Virtual Machine was created and exported with Virtual Box see: The Virtual Machine has been configured with 4GB memory, which is necessary for the OSRM Installation (only OSRM extract process for example the Update of new map data). The machine will work with only 2GB, but more will be necessary for the OSRM Update process. The itixi Ubuntu virtual machine has the following user accounts 2.1 User/Root Username: Password: tixisupport Fahrer2014 To take root account, use sudo or sudo i 2.2 MySQL User Username: Password: root DrivingMaster019 3 Ubuntu Server Installation (PHP5.5 included, MySQL 5.5 included but automatic upgrade to 5.6) Follow the installation instructions Hostname: tixisrv TimeZone: Europe/Zurich 20. August 2014 Seite 3 von 48

5 20. August 2014 Seite 4 von 48

6 Choose Packages (manual package selection not necessary) 20. August 2014 Seite 5 von 48

7 After Installation (VirtualBox + Windows Settings) Get ip: ifconfig Use IP-Address (VirtualBox Internal Network Adapter over NAT) to use NAT-Rules for access. 20. August 2014 Seite 6 von 48

8 SSH Port Translation from localhost to the Virtual Machine internal IP-Address. So we can connect to the SSH Server with: :22 Notice: if you already use a service or application which uses Port 22, use another Port in the field Host-Port -> like and connect then over : Otherwise, the translation it will get blocked or interrupted by that service. 20. August 2014 Seite 7 von 48

9 20. August 2014 Seite 8 von 48

10 4 Mysql 5.6 installation apt-get install mysql-server MySQL root account will be taken from the 5.5 installation 5 Update and Upgrade Update repository and lists sudo apt-get update Upgrade any Ubuntu Server versions sudo apt-get upgrade 20. August 2014 Seite 9 von 48

11 6 Install Webmin (easier to configure server instances) Instructions: User normal user/root to login -> tixisupport 20. August 2014 Seite 10 von 48

12 7 Install PHP Modules 7.1 curl apt-get install php5-curl 7.2 xdebug sudo apt-get install php5-dev php-pear php5-xdebug 20. August 2014 Seite 11 von 48

13 8 Change PHP configuration Default php.ini in /etc/php5/apache2/php.ini sudo nano /etc/php5/apache2/php.ini Go to the end of file xdebug.max_nesting_level= August 2014 Seite 12 von 48

14 9 Install wkhtmltopdf apt-get install wkhtmltopdf Default location of the installed binaries: /usr/bin/wkhtmltopdf 10 Install phpdoc (optional) pear channel-discover pear.phpdoc.org pear install phpdoc/phpdocumentor Dependencies sudo apt-get install php5-xsl sudo apt-get install graphviz 20. August 2014 Seite 13 von 48

15 11 Apache Config Apache Konfigurationen unter /etc/apache2/sites-available/ With a2ensite and a2dissite we can enable and disable virtual server configurations 11.1 Configuration HTTP/Default /etc/apache2/sites-available/tixi.conf Indexes lists the directory, we will not enable that. We will use Symbolic Links in our folderes, so +FollowSymLinks is enabled. <VirtualHost *:80> DocumentRoot /var/www <Directory "/var/www"> allow from all Options -Indexes +FollowSymLinks AllowOverride None </Directory> LogLevel warn ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined </VirtualHost> 20. August 2014 Seite 14 von 48

16 11.2 Directory Within the default web directory we create a symbolic link to the itixi-webdirectory cd /var/www ln -s /srv/tixi/sfitixi/ tixi Same for the OSRM-Website it will be under /srv/osrm-web ln -s /srv/osrm-web/webcontent/ osrm 20. August 2014 Seite 15 von 48

17 11.3 SSL Konfigurieren It ll be necessary to create a default certificate. For productive environment you should buy an official certificate (for example then we can replace the default certificate /etc/apache2/ssl/apache.pem mkdir -p /etc/apache2/ssl openssl req -new -x509 -days 365 -nodes -sha256 -out /etc/apache2/ssl/apache.pem -keyout /etc/apache2/ssl/apache.pem ln -sf /etc/apache2/ssl/apache.pem /etc/apache2/ssl/`/usr/bin/openssl x509 -noout -hash < /etc/apache2/ssl/apache.pem`.0 chmod 600 /etc/apache2/ssl/apache.pem Apache SSL MOD activation a2enmod ssl service apache2 restart Enable ports /etc/apache2/ports.conf SSL Mode Listen 443 should be active # If you just change the port or add more ports here, you will likely also # have to change the VirtualHost statement in # /etc/apache2/sites-enabled/000-default.conf Listen 80 <IfModule ssl_module> </IfModule> Listen August 2014 Seite 16 von 48

18 11.4 Konfiguration HTTPS/SSL /etc/apache2/sites-available/tixi_ssl.conf <VirtualHost *> SSLEngine On SSLCertificateFile /etc/apache2/ssl/apache.pem DocumentRoot /var/www <Directory "/var/www"> allow from all Options -Indexes +FollowSymLinks AllowOverride None </Directory> LogLevel warn ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined </VirtualHost> 20. August 2014 Seite 17 von 48

19 11.5 Apache Webmin The configuration is also possible via Webmin 20. August 2014 Seite 18 von 48

20 20. August 2014 Seite 19 von 48

21 20. August 2014 Seite 20 von 48

22 12 Symfony/sfitixi Install Clone the Symfony project into /srv/tixi/ git clone Symfony components getting all installed by composer (see: ) We can run composer in the project folder /srv/tixi/sfitix/ with php composer.phar To install all dependencies for the first time, just run php composer.phar install Follow the instructions to set parameters 20. August 2014 Seite 21 von 48

23 13 OSRM Install Install dependencies: apt-get install git libboost-dev gcc g++ cmake libstxxl-dev \ libxml2-dev libbz2-dev zlib1g-dev libzip-dev libboost-filesystem-dev \ libboost-thread-dev libboost-system-dev libboost-regex-dev libboost-program-options-dev \ libboost-iostreams-dev libgomp1 libpng12-dev libprotoc8 libprotobuf-dev protobuf-compiler \ liblua5.1-0-dev libluabind-dev pkg-config libosmpbf-dev Get osmconverter wget -O - cc -x c - -lz -O3 -o osmconvert 20. August 2014 Seite 22 von 48

24 13.1 Build OSRM Get OSRM latest stable version mkdir /srv/osrm cd /srv/osrm wget unzip v0.3.7.zip Create stxxl config to specify large data sets /srv/osrm/.stxxl nano.stxxl disk=/tmp/stxxl,2000,syscall build folder /srv/osrm/build mkdir p build cd build cmake.. make Make process will take a while 20. August 2014 Seite 23 von 48

25 Create symlinks in the root folder /srv/osrm cd /srv/osrm ln -s profiles/car.lua profile.lua ln -s build/server.ini server.ini ln -s build/osrm-extract ln -s build/osrm-prepare ln -s build/osrm-routed Change Server.ini /srv/osrm/build/server.ini Threads = 4 IP = hsgrdata=./build/swiss.osrm.hsgr nodesdata=./build/swiss.osrm.nodes edgesdata=./build/swiss.osrm.edges ramindex=./build/swiss.osrm.ramindex fileindex=./build/swiss.osrm.fileindex namesdata=./build/swiss.osrm.names 20. August 2014 Seite 24 von 48

26 13.2 OSRM Extract and Prepare /srv/osrm/build /ROOT-DIR OSRM/ cd build #download latest OSM wget #OSM convert./osmconvert switzerland-latest.osm.pbf -o=swiss.o5m./osmconvert swiss.o5m -o=swiss.osm.pbf #OSM extraction./osrm-extract build/swiss.osm.pbf./osrm-prepare build/swiss.osrm #Start OSRM./osrm-routed Some values if you would like to use more then only one country OSM: August 2014 Seite 25 von 48

27 13.3 OSRM API OSRM Server Requests on Test: OSRM Website mkdir /srv/osrm-web cd /srv/osrm-web git clone mv osrm-frontend/webcontent/. chown R www-data:www-data /srv/osrm-web link that web folder ln s /srv/osrm-web/webcontent /var/www/osrm 20. August 2014 Seite 26 von 48

28 Change configuration nano /srv/osrm-web/webcontent/osrm.config.js change routing and timestamp url ROUTING_ENGINES: [ { url: ' timestamp: ' metric: 0, label: 'ENGINE_0', } 20. August 2014 Seite 27 von 48

29 OSRM Website: OSRM Website will work only without HTTPS/SSL 20. August 2014 Seite 28 von 48

30 14 Virtual Box NAT-Rules 20. August 2014 Seite 29 von 48

31 14.1 SSH with Default-Port: August 2014 Seite 30 von 48

32 14.2 HTTP Website August 2014 Seite 31 von 48

33 14.3 HTTPS Website August 2014 Seite 32 von 48

34 14.4 OSRM API August 2014 Seite 33 von 48

35 14.5 Webmin August 2014 Seite 34 von 48

36 15 Server Scripts All scripts are under: /srv/scripts 15.1 tixi_update_and_rebuild.sh Checks git repository (the respository here is saved in the git.configs under the sfitixi folder) and updates + rebuild server 15.2 tixi_rebuild_server.sh recreates database (run_build) and deploys symfony productive environment 20. August 2014 Seite 35 von 48

37 15.3 osrm_update.sh Stops OSRM process, downloads latest swiss OSM data, extract and convert them, prepare the routings and starts OSRM again with new routing-map data. 20. August 2014 Seite 36 von 48

38 It could be possible to do that all weekend or once per month with a scheduled cron-job (with root account -> sudo i) crontab e For example every saturday at 04:00am 0 4 * * 6 /srv/scripts/osrm_update.sh 20. August 2014 Seite 37 von 48

39 15.4 osrm_start.sh OSRM Server start Wird für den Start auch in den STARTUP des Servers gelegt osrm_routed is execeuted with screen, because the process takes the current user context session. With screen x osrm it is possible to view that session Detach the screen session with (CTRL + A) + D 15.5 osrm_stop.sh Kills the current screen process 20. August 2014 Seite 38 von 48

40 16 MySQL Backup A very good script for MySQL Backups is AutoMYSQLBackup Extracted in /srv/scripts/mysql The install script copies necessary files into the right place. All files are in /srv/sciprts/mysql and executables are in /usr/local/bin/automysqlbackup A config files defines Backup rotation, databases, path and etc. /srv/scripts/mysql/tixi_server.conf A simply script runs the backup /srv/scripts/mysql/ tixi_mysqlbackup.sh Backup path is /srv/backup/db First run creates all subfolders Config defines all databases for backup =() 20. August 2014 Seite 39 von 48

41 All databases are backed up and zipped There is a daily backup, a weekly (saturday) and a monthly (first day of month).. BACKUP ROTATION Daily Backups are rotated weekly. Weekly Backups are run on fridays, unless otherwise specified via CONFIG_do_weekly. Weekly Backups are rotated on a 5 week cycle, unless otherwise specified via CONFIG_rotation_weekly. Monthly Backups are run on the 1st of the month, unless otherwise specified via CONFIG_do_monthly. Monthly Backups are rotated on a 5 month cycle, unless otherwise specified via CONFIG_rotation_monthly. Suggestion: It may be a good idea to copy monthly backups offline or to another server. Create a scheduled cron job 20. August 2014 Seite 40 von 48

42 16.1 Restore.. RESTORING Firstly you will need to uncompress the backup file and decrypt it if encryption was used (see encryption section). eg. gunzip file.gz (or bunzip2 file.bz2) Next you will need to use the mysql client to restore the DB from the sql file. eg. mysql --user=username --pass=password --host=dbserver database < /path/file.sql or mysql --user=username --pass=password --host=dbserver -e "source /path/file.sql" database NOTE: Make sure you use "<" and not ">" in the above command because you are piping the file.sql to mysql and not the other way around. We restore a backup gunzip.sql import 20. August 2014 Seite 41 von 48

43 17 New Client/Mandant To create an new client, separate instance, the following steps are necessary. This will be a new apache virtual server, without developer build websites etc New MySQL database 20. August 2014 Seite 42 von 48

44 Create new MySQL user if necessary and gain access to that database. Default we take the root user. 20. August 2014 Seite 43 von 48

45 17.2 Symfony/sfitixi instance Create an separate folder for the client under /srv/tixi mkdir client_tixizug Clone git repository or copy symfony/sfitixi files in that directory by hand git clone Install dependencies php composer.phar install Set parameters 20. August 2014 Seite 44 von 48

46 Set TIXI parameters nano sfitixi/app/config/tixi.yml It could be possible to create a seperate Google API Account for every client. Change the tixi_parameter_client and client_page for that client instance, and also the admin_mail (for administrator) And if changed, the osrm_server parameter parameters: tixi_parameter_app: itixi tixi_parameter_version: v tixi_parameter_client: Tixi Zug tixi_parameter_client_page: tixi_parameter_files_directory: %kernel.root_dir%/cache/files tixi_parameter_admin_mail: itixizug@gmail.com tixi_parameter_google_apikey: AIzaSyAj6w00ufayAcAv-A8zzLhQspEYflknr4E tixi_parameter_osrm_server: :5000 If parameters changed after build/deploy for productive server, run the symphony deploy again: rm app/cache/* app/logs/* -f -R HTTPDUSER=`ps aux grep -E '[a]pache [h]ttpd [_]www [w]ww-data [n]ginx' grep -v root head -1 cut -d\ -f1` sudo setfacl -R -m u:"$httpduser":rwx -m u:`whoami`:rwx app/cache app/logs sudo setfacl -dr -m u:"$httpduser":rwx -m u:`whoami`:rwx app/cache app/logs php app/console cache:clear --env=prod --no-debug php app/console assetic:dump --env=prod --no-debug chown -R www-data:www-data /srv/tixi/client_tixizug 20. August 2014 Seite 45 von 48

47 run_build or deploy 20. August 2014 Seite 46 von 48

48 17.3 Apache directory Link new folder ln -s /srv/tixi/client_tixizug/sfitixi/web/ /var/www/client_tixizug Make shure apache user can access that folder chown -R www-data:www-data /srv/tixi/client_tixizug Then we can access the new instance with August 2014 Seite 47 von 48

49 18 VirtualBox Import 20. August 2014 Seite 48 von 48