Source Code Management for Continuous Integration and Deployment. Version 1.0 DO NOT DISTRIBUTE

Transcription

1 Source Code Management for Continuous Integration and Deployment Version 1.0

2 Copyright 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. This work may not be reproduced or redistributed, in whole or in part, without prior written permission from Amazon Web Services, Inc. Commercial copying, lending, or selling is prohibited. For corrections or feedback on the course, please us at For all other questions, please us at 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 2

3 Table of Contents Introduction... 4 Overview... 4 Technical Knowledge Prerequisites... 4 Topics Covered... 4 Sign in to the AWS Management Console... 4 Using qwiklabs tm to sign in to the AWS Management Console... 4 Module 1 Source Code Management & Automated Testing... 6 Setting Up Github And Creating Our Own Sample Application and Data Store Repositories... 6 Creating a GitHub account... 6 Forking the sample web application repository... 6 Forking the data sources repository... 7 Connecting to the development environment... 7 Setting up the development environment... 8 Setting up GitHub SSH keys... 8 Telling Git about ourselves Getting the sample Python web application Getting the latest version of the data sources configuration Testing the sample Python web application Additional Resources , 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 3

4 Introduction Overview In this lab you will use a sample Python web application to learn about source code management (SCM) techniques and some common approaches to automated testing. This will then be used as a foundation to build a continuous integration pipeline to automate this using Atlassian Bamboo. Technical Knowledge Prerequisites To successfully complete this lab, you should be familiar with the following: Basic source code management concepts Git as a source code management system Automated application testing concepts, including automated unit testing Python web application programming Topics Covered This lab will take you through source code management techniques and automated testing, including: Forking a version or the sample Python web application on GitHub Using Git as your SCM tool and common operations with Git Unit testing a sample Python web application Test driven development (TDD) concepts Sign in to the AWS Management Console Using qwiklabs tm to sign in to the AWS Management Console Welcome to this self-paced lab! The first step is for you to sign in to Amazon Web Services. 1. To the right of the lab title, click Start Lab. If you are prompted for a token, use the one you received or purchased. Note: A status bar shows the progress of the lab environment creation process. The AWS Management Console is accessible during lab resource creation, but your AWS resources may not be fully available until the process is complete. 2. On the lab details page, notice the lab properties. a. Duration - The time the lab will run before automatically shutting down. b. Setup Time - The estimated time to set up the lab environment. 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 4

5 c. AWS Region - The AWS Region in which the lab resources are created. Note: The AWS Region for your lab will differ depending on your location and the lab setup. 3. In the AWS Management Console section of the qwiklab page, copy the Password to the clipboard. 4. Click the Open Console button. 5. Log into the AWS Management Console using the following steps. a. In the User Name field type awsstudent. b. In the Password field, paste the password copied from the lab details page. c. Click Sign in using our secure server. Note: The AWS account is automatically generated by qwiklab. Also, the login credentials for the awsstudent account are provisioned by qwiklab using AWS Identity Access Management. 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 5

6 Module 1 Source Code Management & Automated Testing In this module we ll ensure that our development environment is configured correctly and ready to use. This will include ensuring that we have Git tools configured, and the correct unit test libraries installed and configured. Because we re using the Flask web application framework for Python, we re going to use some standard approaches to testing Flash applications. This means we ll also be using the unittest test library available for Python. A development environment has been provided containing all the tools and libraries you ll need to work with the example web application and use Git for this lab. We ll be using GitHub to fork a version of the sample Python web application and clone that forked repository to work on it locally. There are lots of options for hosting your source code, both public and private. Many businesses choose to host their source code behind the firewall, and this would work just fine. In fact, Atlassian Bamboo has great support for behind the firewall options like Atlassian Stash or your own Git server. In this lab we use Github for ease of use and to simplify setup and configuration. Setting Up Github And Creating Our Own Sample Application and Data Store Repositories We ll be using GitHub to fork the sample Python web application and also to push our changes to when we ve completed our test and development work. We re also going to fork a GitHub repository that will be responsible for managing the configuration and lifecycle of our relational MySQL database. If you don t have a GitHub account already, you ll need to create one. Even if you do have a GitHub account, you might want to create a new one for the purposes of completing these labs. Creating a GitHub account 1. Browse to 2. Fill out the form to create your personal account 3. Choose a personal plan, e.g. Free ($0/month) 4. Click Finish sign up This will have created a GitHub personal account and logged you in. Forking the sample web application repository To work with the sample Python web application, we ll fork our own version of it. This will let us push any changes we make in subsequent labs to our own GitHub repository. The version of the web application we ll be working with is 1. Open a new browser tab or window go to 2. In the upper right corner click on Fork 3. You could be prompted to select where the repository will be forked. Make sure you select the GitHub account you just created if you are. 4. You will now have a completely new version of the py-flask-signup repository under your GitHub account. 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6

7 Whenever you want to clone this forked repository you can do so, using the HTTPS clone URL on the repository home page, e.g. or the SSH clone URL, e.g. We ll be doing this later in the lab. Forking the data sources repository To create and update our relational database environment for our web application, we ll fork another repository containing a baseline definition of our relational database environment. This repository also includes the database schema used to support our web application. In fact, we ll be continuously delivering schema changes for our database in subsequent labs. The repository we want to fork is 1. Browse to 2. In the upper right corner click on Fork. 3. You could be prompted to select where the repository will be forked. Make sure you select the GitHub account you just created if you are. 4. You will now have a completely new version of the py-flask-signup-datasources repository under your GitHub account. Whenever you want to clone this forked repository you can do so, using the HTTPS clone URL on the repository home page, e.g. or the SSH clone URL, e.g. [email protected]:<your_username>/py-flasksignup.git-datasources.git. We ll do this in subsequent labs. Connecting to the development environment This lab will make use of the development environment that you ve used in previous labs. You will connect to the instance using your local SSH client, typically Putty for Windows or the native SSH client for OSX/Linux. The following steps remind you how to connect to this Development & Build instance. We have created a new Development & Build instance for you, so you ll need to complete the same steps again. If you are comfortable connecting to the Development & Build instance on your own, feel free to skip this section. The development instance will be reachable using its public DNS address. The public DNS of the instance can be found by going to the EC2 Management Console of the AWS Management Console. In your list of running EC2 instances, select the instance with name Linux DevOps Instance to display the instance details. Identify the public DNS value in instance details pane at the bottom of the screen, and copy it to the clipboard. Alternatively you can find the public DNS name of the management instance in the qwiklab environment information screen under the Additional Lab Information section as the value for the DevOpsServerLinuxDNS key. The public DNS name will look something like: ec us-west-2.compute.amazonaws.com Windows SSH clients: Connect to your Development & Build EC2 instance 1. Return to your open browser that has the qwiklab lab information. Download the qwiklabs provided EC2 Key Pair private key file in the PuTTY compatible PPK format. 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 7

8 2. Click on the down arrow next to the Download PEM/PPK drop-down. 3. Click on the Download PPK option 4. Open PuTTY.exe that you downloaded in a previous lab. 5. Enter ec2-user@<your public DNS> into the Host Name field (Ctrl+v). 6. Expand the SSH category by clicking on it. 7. Select the Auth category by clicking on it (not the + symbol next to it). 8. Click Browse and locate the PPK file (ending in.ppk) in your Downloads directory or whatever other location you chose. 9. Click Open. 10. Click Yes when prompted to allow a first connection to this remote SSH server. Since you are using a key pair for authentication, you will not be prompted for a password. Common Issues If PuTTY fails to connect to your EC2 instance, check that: You have entered ec2-user@<hostname> in Putty. You have downloaded the.ppk file for this lab from qwiklab You are using the downloaded.ppk for Private key file for authentication The network you are on allows for outbound TCP connections to destination port 22 OSX & Linux SSH clients: Connect to your Development & Build EC2 instance 1. Return to your open browser that has the qwiklab lab information. Download the qwiklabs provided EC2 Key Pair private key file in the PEM format. 2. Click on the down arrow next to the Download PEM/PPK drop-down. 3. Click on the Download PEM option. 4. Save the file to your Downloads directory, or another directory of your choice. 5. Open the OSX/Linux Terminal application. 6. Enter the below commands substituting the path/filename for the.pem file you downloaded from qwiklabs and pasting ec2-user@<your EC2 hostname> to substitute the example below. chmod 600 ~/Downloads/qwiklab-l pem ssh i ~/Downloads/qwiklab-l pem ec us-west- 2.compute.amazonaws.com Setting up the development environment Once connected to your development environment we re going to create and setup SSH keys for use with our new GitHub account. Setting up GitHub SSH keys We also need to set up our development environment with our GitHub SSH keys and make sure GitHub knows about them. This will mean we can authenticate and push our changes to our newly forked GitHub repository. 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 8

9 GitHub has documentation on how to do this, It s important that we generate SSH keys to be able to authenticate without passwords when we interact with our GitHub account. You can either follow the instructions outlined at or follow the basic steps outlined below in your Development environment: 1. Check for existing SSH keys ls la ~/.ssh You shouldn t see any files with names like id_rsa.pub and id_rsa. This is because we ve created a new development Linux environment for you. If you were doing this on your laptop or another machine you might see keys already created. In that case you could choose to reuse those keys or create new ones for your Github account. 2. Generate a new SSH key, and use the defaults ssh-keygen -t rsa -C [email protected] This will generate a private and public key in ~/.ssh for you. You should see something like: Your identification has been saved in /home/ec2- user/.ssh/id_rsa. Your public key has been saved in /home/ec2- user/.ssh/id_rsa.pub. 3. Now we ll add our new SSH key to our Github account. To do this you ll need to copy the content of the public key to your clipboard. To do this: cat ~/.ssh/id_rsa.pub which will display the contents of your public key. Copy the contents from the beginning of the key starting with ssh-rsa to the end of the address you entered, e.g. [email protected]. Make sure not to include any other characters or lines. 4. Now we add the key to our Github account. To do this: a. Log in to Github.com and click on the top right gear icon, Settings b. On the left hand navigation click on SSH keys c. Click the Add SSH key button d. In Title field enter something like AWS lab development environment e. In the Key field paste the contents of your public key copied in the previous setp f. Click Add key Once you ve configured GitHub with your SSH keys, you should be able to run: $ ssh -T [email protected] 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 9

10 Hi <username>! You've successfully authenticated, but GitHub does not provide shell access. Telling Git about ourselves We re going to tell our local Git install about ourselves as well. This will help identify ourselves when we push changes to our remote repository on Github too. To do this we want to run the following commands: git config --global user.name "Your Name" git config --global user. Git will store this information in our ~/.gitconfig file in our development environment. Getting the sample Python web application 1. Now that we have configured our Git environment and can successfully authenticate against our new GitHub account, we ll get the latest version of our sample Python application. 2. We have our own version of the forked sample application located in a new public Github repository. On your development environment, a. Make a directory in your home directory called git and change to that directory mkdir $HOME/git cd $HOME/git b. Get the latest version of the sample web application. To do this you ll need to find the SSH clone URL for your GitHub repository. To do this, browse to Github.com and your py-flask-signup repository. On the right hand side you ll see a navigation bar like below: Click on the SSH link and copy the SSH clone URL into your clipboard. Once you ve done that go back to your development environment and clone your 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 10

11 fork of the same Python web application git repository. E.g., git clone c. You should see a new directory, py-flask-signup. Change to that directory. cd py-flask-signup 3. We want to change to a branch that has the latest application changes in it, called agecollection'. To do this on the CLI, git checkout age-collection 4. If you list all git branches in your repository now you should see the micro-blog branch, and it should have an asterisk next to it as the current branch. E.g. git branch * age-collection master Getting the latest version of the data sources configuration 1. From your $HOME/git directory, get the latest version of your data sources repository. As you did for the sample web application, retrieve the SSH clone URL for your data sources forked repository and clone it, e.g. git clone [email protected]:<your_username>/py-flask-signupdatasources.git 2. You should see a new directory, py-flask-signup-datasources. Change to that directory. cd py-flask-signup-datasources 3. Change to the age-collection branch, e.g. git checkout age-collection git branch * age-collection master 4. Configure the remote repository, e.g. git config --global push.default simple 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 11

12 git push --set-upstream origin age-collection 5. You should now have a cloned version of the py-flask-signup-datasources repository inside your git working area. We ll use this for a later lab. Testing the sample Python web application 1. If you inspect the contents of the py-flask-signup directory you ll find our web application along with lots of other supporting configuration. The part we re most interested in is the test suite. This is located in the tests/ directory. a. We ll run our tests from this directory, and use the Python unittest library. To run your test case, run python tests/application-tests.py. b. Try running the test harness, python tests/application-tests.py. You should now see a Python import error, pointing to the fact that our application module can t be loaded. Something like: ImportError: No module named application To fix this error we need to export an environment variable, $PYTHONPATH c. To do this, type the following in your bash shell: export PYTHONPATH=/home/ec2-user/git/py-flask-signup If you now echo that environment variable you should see your working directory for the cloned py-flask-signup web application repository echo $PYTHONPATH /home/ec2-user/git/py-flask-signup We ll need to set this environment variable later on when we automate running these tests from our CI environment. d. Try running the test harness again, python tests/application-tests.py. You should now see a Python module import error similar to, ImportError: No module named flask This message tells us that our development environment doesn t have the necessary Python modules installed. In fact, we re missing at least two modules for our Flask application, e. To fix this, let s install the Python package manager, pip, and then use pip to install the missing Flask modules. Run the following: sudo yum install python26-pip sudo pip install Flask Flask-SQLAlchemy When we come to doing the work to automate the creation of our deployable application artifact in a later lab, we ll have to bear this in mind as well. f. Finally, try running the test harness again, python tests/application-tests.py. You should see something like the following: 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 12

13 python tests/application-tests.py Ran 5 tests in 0.016s OK Inspect the output. You should see tests run successfully, i.e. OK. In a test driven development model, you would often see failing tests until bugs are fixes, or features are built against tests. We re not trying to simulate test driven development here, just to point out the importance of having a test harness for your applications, how you might go about running it. The main goal will be to automate this test effort in subsequent labs, so we always test our code on every build and deploy. 2. Open tests/application-tests.py. Have a look at all the methods starting with test_. These are our tests in the test case we re running. 3. Have a think about other tests you d like to run. If you have time and are ambitious try implementing some of your own custom tests. 4. If you made any changes to the application or test harness, and making sure that you have an application that passes all the tests you can commit your changes. This will also version them for you and give you an audit trail. To check which files you ve modified since working on your test harness run: git status This will show you the status of your local Git repository. You should for example see something like: modified: modified: application.py We ll stage these files to be committed: tests/application-tests.py git add application.py tests/application-tests.py And then commit them: git commit m Bug fixes to ensure all tests pass successfully. 5. The final task we want to do perform is push our changes back to the remote Github repository. Git makes this easy provided that we have the right permissions and have set up our SSH keys properly at the beginning of this lab. 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 13

14 To do this, run these commands: git config --global push.default simple git push --set-upstream origin age-collection You should see something like the following output if successful: Total 0 (delta 0), reused 0 (delta 0) To [email protected]:aws-tools/py-flask-signup.git * [new branch] age-collection -> age-collection Branch micro-blog set up to track remote branch age-collection from origin. Congratulations! You ve now implemented some basic source code management and built an automated test harness for your Flask application using the unittest library. Having good development processes will help improve the quality of the software you re building and delivering to your customers. In the next lab we ll automate this process and build it into a continuous integration pipeline. Additional Resources AWS Training and Certification. For feedback, suggestions, or corrections, please [email protected] 2013, 2014 Amazon Web Services, Inc. and its affiliates. All rights reserved. 14