Using Toaster in a Production Environment

Transcription

1 Using Toaster in a Production Environment Alexandru Damian, David Reyna, Belén Barros Pena Yocto Project Developer Day ELCE 17 Oct 2014

2 Introduction Agenda: What is Toaster Toaster out of the box Toaster in a production environment Comparison of database servers Comparison of web servers Toaster: interaction with auto build and auto test tools How-to guide of a best practice configuration Toaster in the real world

3 What is Toaster The new UI interface for the Yocto Project Project and management isolation and scalability System scalability, remote management

4 Toaster out of the box Toaster provides a simple out-of-box configuration, where a few commands will automatically set up an sqlite database and start a Django web server. This default setup is sufficient to immediately support a small number of users and projects, before you move on to the more robust and scalable solutions. Host Project sqlite Bitbake Toaster Django Server Browser

5 How-to: Starting Toaster out of the box Set up the project Start Toaster data collection and web server Build the project Open Toaster in browser and see results $ source poky/oe-init-build-env $ source toaster start $ bitbake core-image-minimal $ xdg-open $

6 Toaster in a production environment

7 Advantages of Production Environment for Toaster Multiple clients Multiple build machines Scalability and reliability Ability for long-term data trending analysis Build Blade(s) Build Host Projects Bitbake Toaster DB Host sql Web Host Web Server Client s Browser

8 Review of Database Servers Sqlite3 default server Advantages: No configuration needed, easy installation Emphasizes economy, efficiency, independence, and simplicity for data storage local applications and devices Disadvantages: No remote host access, slow, no transactional support MySQL or PostgreSQL server Advantages: Advanced SQL features, fast, reliable Emphasizes scalability, concurrency, centralization, and control Disadvantages: Some more complex setup, Need a good database administrator to tune it out

9 Review of Web Servers Django development server Advantages: Available with out-of-box Django installations No configuration needed Disadvantages: Very poor performance, no customization, security issues Apache or nginx server Advantages: Very fast, very configurable Disadvantages: Needs complex configurations, and a system administrator to look after it

10 Toaster and autobuilders/auto-testers Toaster is not a replacement for your continuous integration software Toaster interacts with your continuous integration software Toaster provides an interface that is customized and fine-tuned to the capabilities (and quirks) of the Yocto Project build system Which means an easier way to configure and debug your Yocto Project builds

11 How-to Guide Setting up a Toaster production environment with MySQL and Apache

12 Checkout toaster for server use Select location for production toaster, and checkout bitbake Install all dependencies Make it reachable by the Apache user $ mkdir /opt/ && cd /opt/ $ git clone git://git.openembedded.org/bitbake #production...or for the test-to-merge branch $ git clone -b toaster/install_test --single-branch \ bitbake $ pip install -r /opt/bitbake/toaster-requirements.txt $ chgrp -R www-data: /opt/bitbake/lib/toaster/

13 Configuring MySQL Install mysql-server Create a sql user account, and database with minimal rights (not like in the example) Update bitbake/lib/toaster/toastermain/settings.py root$ apt-get install mysql-server root$ mysql -u root mysql> CREATE USER 'newuser'@'localhost' IDENTIFIED BY 'password'; mysql> GRANT ALL PRIVILEGES ON *. * TO 'newuser'@'localhost'; mysql> GRANT ALL PRIVILEGES ON *. * TO 'newuser'@'localhost'; mysql> CREATE DATABASE 'toaster'; root$ cat /opt/bitbake/lib/toaster/toastermain/settings.py DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'toaster', 'USER': 'newuser', 'PASSWORD': 'password', 'HOST': ' ', 'PORT': '3306', }

14 Configuring Apache Install Apache and mod_wsgi Configure Apache to load Toaster root$ apt-get install apache2 libapache2-mod-wsgi root$ a2enmod wsgi root$ cat /etc/apache2/sites-available/000-default.conf # the WSGIPythonPath is global WSGIPythonPath /opt/bitbake/lib/toaster/ #snip in VirtualHost WSGIScriptAlias / /opt/bitbake/lib/toaster/toastermain/wsgi.py <Directory //opt/bitbake/lib/toaster/toastermain/> <Files wsgi.py> Require all granted </Files> </Directory>

15 Configuring Apache (continued) Collect static media from Toaster Configure Apache to serve static media $ mkdir /var/www/html/static && cd /var/www/html/static $ /opt/bitbake/lib/toaster/manage.py collectstatic $ cat /etc/apache2/sites-available/000-default.conf # in VirtualHost, AHEAD of the WSGIScriptAlias definition Alias /static/ /var/www/html/static/ <Directory /var/www/html/static/> Require all granted </Directory> WSGIScriptAlias / /opt/bitbake/lib/toaster/toastermain/wsgi.py

16 Starting up Toaster Sync databases for toaster Start apache root$ /opt/bitbake/lib/toaster/manage.py syncdb root$ /opt/bitbake/lib/toaster/manage.py migrate orm root$ service apache2 restart # when happy with the toaster settings.py, # save it in your local git: root$ cd /opt/bitbake root$ git commit -p -m Local custom settings.py

17 Use Toaster to log your builds Configure Database settings in your source tree settings.py, build and see results # set database in the poky source tree to point to the same database instance user$ cat ~/poky/bitbake/lib/toaster/toastermain/settings.py DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'toaster', 'USER': 'newuser', 'PASSWORD': 'password', 'HOST': ' ', 'PORT': '3306', } # build as normal with toaster user$ source oe-init-build-env user$ source toaster start user$ bitbake core-image-minimal # open up the web interface to see your builds user$ xdg-open

18 Some Big Gotchas Under wsgi, all Python environment (like libs, interpreters, etc) are shared on the same instance of Apache. This may lead to some very un-intuitive results if you want to run two different django apps on the same apache server. The recommended way to multiplex django apps under a single server is run each Django app in its own apache instance, and provide a single routing point using a reverse-proxy nginx server. Apache/wsgi aggressively caches the.py files in the application, in other words if you make changes to your application they won't be picked up on the fly like with the Django development server, so you need to restart apache. Neither the database nor the queries have been tuned for the high workloads such as on a build farm. It may work out of the box, it may not, so further testing is needed.

19 Toaster in the Real World

20 Real world uses of Toaster Wind River has been using Toaster for the last 6 months in production environments. Lessons learned include: The toaster captured build event data was being clobbered if an eventhandler was defined in any bitbake class that didn't have an eventmask We found that the 'best practice' to provide an event mask for each event handler (which filtered the 'right' events being sent to an event handler) was not always implemented in Wind River bbclasses A race condition was discovered when starting bitbake server, observer, and a build of a specific target when the launches were split out of the original upstream script Wind River is moving Toaster into its build farm: Leverage Toaster s HTTP-based remote interface Leverage Toaster s error capture and presentation Leverage new insights from the big data of 10,000+ build logs

21 Toaster : The plan

22 An IDE for distribution engineers? Understand the build process (analysis) Configure and start builds (the basics) DONE IN PROGRESS Modify create image recipes (customization) Share metadata, configuration and artifacts (collaboration) Edit and patch source (development) Layer sources, error collection and analysis, presentation of performance benchmarks and test results, whatever else you want (additional goodies)

23 The goal An easy way to deal with the Yocto Project build system, and a controlled and safe configuration environment for your customers and developers (i.e. a build system fit for humans)

24 Questions and Answers

25 Thank you for your participation!

26