How To Manage An Openstack Cloud On A Raspberry Pommer (Rpc) V9 Software (Rp) V2.5 (Rper) (Rproper) V3.5.5 And V3 (Rpl

Transcription

1 rackspace.com/cloud/private

2 Rackspace Private Cloud v9 Installation RPC v9.0 ( ) Copyright 2015 Rackspace All rights reserved. This documentation is intended for Rackspace customers who are interested in installing an Open- Stack-powered private cloud according to the recommendations of Rackspace. ii

3 Table of Contents 1. Preface About Rackspace Private Cloud Software Rackspace Private Cloud configuration Rackspace Private Cloud support Overview Ansible Linux Containers (LXC) Host layout Host networking OpenStack Networking Installation requirements Installation workflow Deployment host Installing the operating system Configuring the operating system Installing source and dependencies Configuring Secure Shell (SSH) keys Target hosts Installing the operating system Configuring Secure Shell (SSH) keys Configuring the operating system Configuring LVM Configuring the network Reference architecture Configuring the network on a target host Deployment configuration Prerequisites Configuring target host networking Configuring target hosts Configuring service passwords Configuring the hypervisor (optional) Configuring the Image Service (optional) Configuring the Block Storage service (optional) Configuring Block Storage Service for multiple NetApp backends (optional) Configure the Block Storage Service with NFS protocols (optional) Creating Block Storage availability zones (optional) Foundation playbooks Running the foundation playbook Troubleshooting Infrastructure playbooks Running the infrastructure playbook Verifying infrastructure operation OpenStack playbooks Utility Container Overview Running the OpenStack playbook Verifying OpenStack operation Rackspace Private Cloud monitoring iii

4 9.1. Service and response Hardware monitoring Software monitoring CDM monitoring Running monitoring playbooks Operations Adding a compute host Galera cluster maintenance Removing nodes Starting a cluster Galera cluster recovery Single-node failure Multi-node failure Complete failure Restoring from backup after a complete failure Rebuilding a container Additional resources Document change history iv

5 List of Figures 2.1. Host Layout Overview Network components Container network architecture Bare/Metal network architecture Networking agents containers Compute hosts Installation workflow Installation workflow Installation workflow Target hosts for infrastructure, networking, and storage services Target hosts for Compute service Installation workflow Installation workflow Installation workflow Installation workflow v

6 1. Preface Rackspace Private Cloud Software has been developed by Rackspace as a way to quickly install an OpenStack private cloud, configured as recommended by Rackspace OpenStack specialists About Rackspace Private Cloud Software Rackspace Private Cloud Software uses Ansible to create an OpenStack cluster on Ubuntu Linux. The installation process provides a familiar approach for Linux system administrators, and the environment can be updated easily without downloading and installing a new ISO Rackspace Private Cloud configuration Rackspace Private Cloud Software uses Ansible and Linux Containers (LXC) to install and manage OpenStack Icehouse with the following services: Identity (keystone) Image Service (glance) Compute (nova) Networking (neutron) Block Storage (cinder) Orchestration (heat) Dashboard (horizon) RPC also provides the following infrastructure, monitoring, and logging services to support OpenStack: Galera with MariaDB RabbitMQ Memcached Rsyslog Logstash Elasticsearch with Kibana 1.3. Rackspace Private Cloud support Rackspace offers 365x24x7 support for Rackspace Private Cloud Software. If you are interested in purchasing Escalation Support or Core Support for your cloud, or taking advantage of our training offerings, contact us at: <opencloudinfo@rackspace.com>. 1

7 You can also visit the RPC community forums. The forum is open to all RPC users and is moderated and maintained by Rackspace personnel and OpenStack specialists: For more information about Rackspace Private Cloud, please visit the Rackspace Private Cloud pages: Software and Reference Architecture Support Resources For any other information regarding Rackspace Private Cloud Software, refer to the Rackspace Private Cloud release notes. 2

8 2. Overview Rackspace Private Cloud (RPC) v9 Software uses a combination of Ansible and Linux Containers (LXC) to install and manage OpenStack Icehouse. This chapter discusses the following topics: The technology used by Rackspace Private Cloud Software The environment and network architecture Requirements to install Rackspace Private Cloud Software The installation process workflow 2.1. Ansible RPC v9 Software uses a combination of Ansible and Linux Containers (LXC) to install and manage OpenStack Icehouse. Ansible provides an automation platform to simplify system and application deployment. Ansible manages systems using Secure Shell (SSH) instead of unique protocols that require remote daemons or agents. Ansible uses playbooks written in the YAML language for orchestration. For more information, see Ansible - Intro to Playbooks. In this guide, Rackspace refers to the host running Ansible playbooks as the deployment host and the hosts on which Ansible installs RPC as the target hosts. A recommended layout for installing RPC involves five target hosts in total: three infrastructure hosts, one compute host, and one logging host. RPC software also supports one or more optional storage hosts. All hosts require at least four 10 Gbps network interfaces. In Rackspace datacenters, hosts can use an additional 1 Gbps network interface for service network access. More information on setting up target hosts can be found in Section 2.3, Host layout [4]. For more information on physical, logical, and virtual network interfaces within hosts see Section 2.4, Host networking [6] Linux Containers (LXC) Containers provide operating-system level virtualization by enhancing the concept of chroot environments, which isolate resources and file systems for a particular group of processes without the overhead and complexity of virtual machines. They access the same kernel, devices, and file systems on the underlying host and provide a thin operational layer built around a set of rules. The Linux Containers (LXC) project implements operating system level virtualization on Linux using kernel namespaces and includes the following features: Resource isolation including CPU, memory, block I/O, and network using cgroups. 3

9 Selective connectivity to physical and virtual network devices on the underlying physical host. Support for a variety of backing stores including LVM. Built on a foundation of stable Linux technologies with an active development and support community. Useful commands: List containers and summary information such as operational state and network configuration: # lxc-ls --fancy Show container details including operational state, resource utilization, and veth pairs: # lxc-info --name container_name Start a container: # lxc-start --name container_name Attach to a container: # lxc-attach --name container_name Stop a container: # lxc-stop --name container_name 2.3. Host layout The recommended layout contains a minimum of five hosts (or servers). Three infrastructure hosts One compute host One logging host To use the optional Block Storage (cinder) service, a sixth host is required. Block Storage hosts require an LVM volume group named cinder-volumes. See Section 2.6, Installation requirements [13] and Section 4.4, Configuring LVM [18] for more information. The hosts are called target hosts because Ansible deploys the RPC environment within these hosts. The RPC environment also requires a deployment host from which Ansible orchestrates the deployment process. One of the target hosts can function as the deployment host. 4

10 At least one hardware load balancer must be included to manage the traffic among the target hosts. Infrastructure target hosts contain the following services: Infrastructure: Galera RabbitMQ Memcached Logging OpenStack: Identity (keystone) Image Service (glance) Compute management (nova) Networking (neutron) Orchestration (heat) Dashboard (horizon) Compute target hosts contain the following services: Compute virtualization Logging Logging target hosts contain the following services: Rsyslog Logstash Elasticsearch with Kibana (Optional) Storage target hosts contain the following services: Block Storage scheduler Block Storage volumes 5

11 Figure 2.1. Host Layout Overview 2.4. Host networking The combination of containers and flexible deployment options requires implementation of advanced Linux networking features such as bridges and namespaces. Bridges provide layer 2 connectivity (similar to switches) among physical, logical, and virtual network interfaces within a host. After creating a bridge, the network interfaces are virtually "plugged in" to it. RPC software uses bridges to connect physical and logical network interfaces on the host to virtual network interfaces within containers on the host. Namespaces provide logically separate layer 3 environments (similar to routers) within a host. Namespaces use virtual interfaces to connect with other namespaces including the host namespace. These interfaces, often called veth pairs, are virtually "plugged in" between namespaces similar to patch cables connecting physical devices such as switches and routers. Each container has a namespace that connects to the host namespace with one or more veth pairs. Unless specified, the system generates random names for veth pairs. The relationship between physical interfaces, logical interfaces, bridges, and virtual interfaces within containers is shown in Figure 2.2, Network components [7]. 6

12 Figure 2.2. Network components Target hosts can contain the following network bridges: LXC internal lxcbr0: Mandatory (automatic) Provides external (typically internet) connectivity to containers. Automatically created and managed by LXC. Does not directly attach to any physical or logical interfaces on the host because iptables handles connectivity. Attaches to eth0 in each container. Container management br-mgmt: Mandatory Provides management of and communication among infrastructure and OpenStack services. Manually created and attaches to a physical or logical interface, typically a bond0 VLAN subinterface. Also attaches to eth1 in each container. Storage br-storage: Optional 7

13 Provides segregated access to block storage devices between Compute and Block Storage hosts. Manually created and attaches to a physical or logical interface, typically a bond0 VLAN subinterface. Also attaches to eth2 in each associated container. OpenStack Networking tunnel/overlay br-vxlan: Mandatory Provides infrastructure for VXLAN tunnel/overlay networks. Manually created and attaches to a physical or logical interface, typically a bond1 VLAN subinterface. Also attaches to eth10 in each associated container. OpenStack Networking provider br-vlan: Mandatory Provides infrastructure for VLAN and flat networks. Manually created and attaches to a physical or logical interface, typically bond1. Also attaches to eth11 in each associated container. Does not contain an IP address because it only handles layer 2 connectivity. Figure 2.3, Container network architecture [9] provides a visual representation of network components for services in containers. 8

14 Figure 2.3. Container network architecture The RPC software installs the Compute service in a bare metal environment rather than within a container. Figure 2.4, Bare/Metal network architecture [10] provides a visual representation of the unique layout of network components on a Compute host. 9

15 Figure 2.4. Bare/Metal network architecture 2.5. OpenStack Networking OpenStack Networking (neutron) is configured to use a DHCP agent, L3 Agent and Linux Bridge agent within a networking agents container. Figure 2.5, Networking agents containers [11] shows the interaction of these agents, network components, and connection to a physical network. 10

16 Figure 2.5. Networking agents containers 11

17 The Compute service uses the KVM hypervisor. Figure 2.6, Compute hosts [12] shows the interaction of instances, Linux Bridge agent, network components, and connection to a physical network. Figure 2.6. Compute hosts 12

18 2.6. Installation requirements Deployment host: Required items: Ubuntu LTS (Trusty Tahr) or compatible operating system that meets all other requirements. Secure Shell (SSH) client supporting public key authentication. Synchronized network time (NTP) client. Python 2.7 or later. Target hosts: Required items: Ubuntu Server LTS (Trusty Tahr) 64-bit operating system, with Linux kernel version generic or later. SSH server supporting public key authentication. Synchronized NTP client. Optional items: For hosts providing Block Storage (cinder) service volumes, a Logical Volume Manager (LVM) volume group named cinder-volumes. LVM volume group named lxc to store container file systems. If the lxc volume group does not exist, containers in the root file system will automatically be installed. Note 2.7. Installation workflow Each container creates a 5 GB logical volume. Plan storage accordingly to support the quantity of containers on each target host. This diagram shows the general workflow associated with RPC installation. 13

19 Figure 2.7. Installation workflow 14

20 3. Deployment host Figure 3.1. Installation workflow The RPC software installation process requires one deployment host. The deployment host contains Ansible and orchestrates the RPC installation on the target hosts. One of the target hosts, preferably one of the infrastructure variants, can be used as the deployment host. To use a deployment host as a target host, follow the steps in Chapter 4, Target hosts [17] on the deployment host. This guide assumes separate deployment and target hosts Installing the operating system Install the Ubuntu Server (Trusty Tahr) LTS 64-bit operating system on the deployment host with at least one network interface configured to access the Internet or suitable local repositories Configuring the operating system Install additional software packages and configure NTP. 1. Install additional software packages if not already installed during operating system installation: # apt-get install aptitude build-essential git ntp ntpdate \ openssh-server python-dev sudo 2. Configure NTP to synchronize with a suitable time source Installing source and dependencies Install the source and dependencies for the deployment host. 1. Clone the repository into the /opt directory: 15

21 # cd /opt # git clone -b TAG Replace TAG with the current stable release tag. 2. Install pip and dependencies: # curl -O # python get-pip.py \ --find-links=" python_packages/icehouse" \ --no-index 3. Install Ansible and dependencies: # pip install -r /opt/openstack-ansible/requirements.txt Note The command described above will install the correct version of Ansible. Do not use the apt package manager. If Ansible is installed using the latter, uninstall it before performing this step Configuring Secure Shell (SSH) keys Ansible uses Secure Shell (SSH) with public key authentication for connectivity between the deployment and target hosts. To reduce user interaction during Ansible operations, key pairs should not include passphrases. However, if a passphrase is required, consider using the ssh-agent and ssh-add commands to temporarily store the passphrase before performing Ansible operations. 16

22 4. Target hosts Figure 4.1. Installation workflow The RPC software installation process requires at least five target hosts that will contain the OpenStack environment and supporting infrastructure. On each target host, perform the following tasks: Naming target hosts. Install the operating system. Generate and set up security measures. Update the operating system and install additional software packages. Create LVM volume groups. Configure networking devices Installing the operating system Install the Ubuntu Server (Trusty Tahr) LTS 64-bit operating system on the target host with at least one network interface configured to access the Internet or suitable local repositories. Note On target hosts without local (console) access, Rackspace recommends adding the Secure Shell (SSH) server packages to the installation Configuring Secure Shell (SSH) keys Ansible uses Secure Shell (SSH) for connectivity between the deployment and target hosts. 1. Copy the contents of the public key file on the deployment host to the /root/.ssh/ authorized_keys on each target host. 2. Test public key authentication from the deployment host to each target host. SSH should provide a shell without asking for a password. 17

23 4.3. Configuring the operating system Check the kernel version, install additional software packages, and configure NTP. 1. Check the kernel version. It should be generic or later. 2. Install additional software packages if not already installed during operating system installation: # apt-get install bridge-utils debootstrap ifenslave lsof \ lvm2 ntp ntpdate openssh-server sudo tcpdump vlan 3. Add the appropriate kernel modules to the etc/modules file to enable VLAN and bond interfaces: # echo 'bonding' >> /etc/modules # echo '8021q' >> /etc/modules 4. Configure NTP to synchronize with a suitable time source. 5. Reboot the host to activate the changes Configuring LVM 1. To use the optional Block Storage (cinder) service, create an LVM volume group named cinder-volumes on the Block Storage host. A metadata size of 2048 must be specified during physical volume creation. For example: # pvcreate --metadatasize 2048 physical_volume_device_path # vgcreate cinder-volumes physical_volume_device_path 2. Optionally, create an LVM volume group named lxc for container file systems. If the lxc volume group does not exist, containers will be automatically installed into the file system under /var/lib/lxc by default Configuring the network Although Ansible automates most deployment operations, networking on target hosts requires manual configuration because it can vary dramatically per environment. For demonstration purposes, these instructions use a reference architecture with example network interface names, networks, and IP addresses. Modify these values as needed for the particular environment. The reference architecture for target hosts contains the following mandatory components: A bond0 interface using two physical interfaces. For redundancy purposes, avoid using more than one port on network interface cards containing multiple ports. The example configuration uses eth0 and eth2. Actual interface names can vary depending on hardware and drivers. Configure the bond0 interface with a static IP address on the host management network. 18

24 A bond1 interface using two physical interfaces. For redundancy purposes, avoid using more than one port on network interface cards containing multiple ports. The example configuration uses eth1 and eth3. Actual interface names can vary depending on hardware and drivers. Configure the bond1 interface without an IP address. Container management network subinterface on the bond0 interface and br-mgmt bridge with a static IP address. The OpenStack Networking VXLAN subinterface on the bond1 interface and br-vxlan bridge with a static IP address. The OpenStack Networking VLAN br-vlan bridge on the bond1 interface without an IP address. The reference architecture for target hosts can also contain the following optional components: Storage network subinterface on the bond0 interface and br-storage bridge with a static IP address. For more information, see OpenStack Ansible Networking Reference architecture After establishing initial host management network connectivity using the bond0 interface, modify the /etc/network/interfaces file as described in the following procedure. Procedure 4.1. Modifying the network interfaces file 1. Physical interfaces: # Physical interface 1 auto eth0 iface eth0 inet manual bond-master bond0 bond-primary eth0 # Physical interface 2 auto eth1 iface eth1 inet manual bond-master bond1 bond-primary eth1 # Physical interface 3 auto eth2 iface eth2 inet manual bond-master bond0 # Physical interface 4 auto eth3 iface eth3 inet manual bond-master bond1 2. Bonding interfaces: # Bond interface 0 (physical interfaces 1 and 3) auto bond0 iface bond0 inet static 19

25 bond-slaves none bond-mode active-backup bond-miimon 100 bond-downdelay 200 bond-updelay 200 address HOST_IP_ADDRESS netmask HOST_NETMASK gateway HOST_GATEWAY dns-nameservers HOST_DNS_SERVERS # Bond interface 1 (physical interfaces 2 and 4) auto bond1 iface bond1 inet manual bond-slaves none bond-mode active-backup bond-miimon 100 bond-downdelay 250 bond-updelay 250 If not already complete, replace HOST_IP_ADDRESS, HOST_NETMASK, HOST_GATEWAY, and HOST_DNS_SERVERS with the appropriate configuration for the host management network. 3. Logical (VLAN) interfaces: # Container management VLAN interface iface bond0.container_mgmt_vlan_id inet manual vlan-raw-device bond0 # OpenStack Networking VXLAN (tunnel/overlay) VLAN interface iface bond1.tunnel_vlan_id inet manual vlan-raw-device bond1 # Storage network VLAN interface (optional) iface bond0.storage_vlan_id inet manual vlan-raw-device bond0 Replace *_VLAN_ID with the appropriate configuration for the environment. 4. Bridge devices: # Container management bridge auto br-mgmt iface br-mgmt inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references tagged interface bridge_ports bond0.container_mgmt_vlan_id address CONTAINER_MGMT_BRIDGE_IP_ADDRESS netmask CONTAINER_MGMT_BRIDGE_NETMASK dns-nameservers CONTAINER_MGMT_BRIDGE_DNS_SERVERS # OpenStack Networking VXLAN (tunnel/overlay) bridge auto br-vxlan iface br-vxlan inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references tagged interface 20

26 bridge_ports bond1.tunnel_vlan_id address TUNNEL_BRIDGE_IP_ADDRESS netmask TUNNEL_BRIDGE_NETMASK # OpenStack Networking VLAN bridge auto br-vlan iface br-vlan inet manual bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references untagged interface bridge_ports bond1 # Storage bridge (optional) auto br-storage iface br-storage inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port reference tagged interface bridge_ports bond0.storage_vlan_id address STORAGE_BRIDGE_IP_ADDRESS netmask STORAGE_BRIDGE_NETMASK Replace *_VLAN_ID, *_BRIDGE_IP_ADDRESS, and *_BRIDGE_NETMASK, *_BRIDGE_DNS_SERVERS with the appropriate configuration for the environment Configuring the network on a target host This example uses the following parameters to configure networking on a single target host. The sample interface configurations are intended to illustrate the scope and design of the network. When deploying outside of Rackspace data centers, settings should be configured according to the data center's network hardware. See Figure 4.2, Target hosts for infrastructure, networking, and storage services [23] and Figure 4.3, Target hosts for Compute service [24] for a visual representation of these parameters in the architecture. VLANs: Host management: Untagged/Native Container management: 10 Tunnels: 30 Storage: 20 Networks: Host management: /22 Container management: /22 Tunnel: /22 Storage: /22 21

27 Addresses: Host management: Host management gateway: DNS servers: Container management: Tunnel: Storage:

28 Figure 4.2. Target hosts for infrastructure, networking, and storage services 23

29 Figure 4.3. Target hosts for Compute service Contents of the /etc/network/interfaces file: # Physical interface 1 auto eth0 iface eth0 inet manual bond-master bond0 bond-primary eth0 # Physical interface 2 auto eth1 iface eth1 inet manual bond-master bond1 bond-primary eth1 # Physical interface 3 auto eth2 iface eth2 inet manual bond-master bond0 # Physical interface 4 auto eth3 iface eth3 inet manual bond-master bond1 24

30 # Bond interface 0 (physical interfaces 1 and 3) auto bond0 iface bond0 inet static bond-slaves none bond-mode active-backup bond-miimon 100 bond-downdelay 200 bond-updelay 200 address netmask gateway dns-nameservers # Bond interface 1 (physical interfaces 2 and 4) auto bond1 iface bond1 inet manual bond-slaves none bond-mode active-backup bond-miimon 100 bond-downdelay 250 bond-updelay 250 # Container management VLAN interface iface bond0.10 inet manual vlan-raw-device bond0 # OpenStack Networking VXLAN (tunnel/overlay) VLAN interface iface bond1.30 inet manual vlan-raw-device bond1 # Storage network VLAN interface (optional) iface bond0.20 inet manual vlan-raw-device bond0 # Container management bridge auto br-mgmt iface br-mgmt inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references tagged interface bridge_ports bond0.10 address netmask dns-nameservers # OpenStack Networking VXLAN (tunnel/overlay) bridge auto br-vxlan iface br-vxlan inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references tagged interface bridge_ports bond1.30 address netmask # OpenStack Networking VLAN bridge auto br-vlan iface br-vlan inet manual 25

31 bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port references untagged interface bridge_ports bond1 # Storage bridge (optional) auto br-storage iface br-storage inet static bridge_stp off bridge_waitport 0 bridge_fd 0 # Bridge port reference tagged interface bridge_ports bond0.20 address netmask In non-rackspace data centers, the service network configuration should be commented out of /etc/rpc_deploy/rpc_user_config.yml. The commented-out section should appear as follows: # Cidr used in the Service network # snet: /22 #- network: # group_binds: # - glance_api # - nova_compute # - neutron_linuxbridge_agent # type: "raw" # container_bridge: "br-snet" # container_interface: "eth3" # ip_from_q: "snet" 26

32 5. Deployment configuration Figure 5.1. Installation workflow Ansible references a handful of files containing mandatory and optional configuration directives. These files must be modified to define the target environment before running the Ansible playbooks. Perform the following tasks: Configure Target host networking to define bridge interfaces and networks. Configure a list of target hosts on which to install the software. Configure virtual and physical network relationships for OpenStack Networking (neutron). Configure passwords for all services. (Optional) Configure the hypervisor. (Optional) Configure Block Storage (cinder) to use NetApp backends. (Optional) Create Block Storage availability zones Prerequisites Copy the contents of the /opt/openstack-ansible/etc/rpc_deploy directory to the /etc/rpc_deploy directory. # cp -R /opt/openstack-ansible/etc/rpc_deploy /etc 5.2. Configuring target host networking Modify the /etc/rpc_deploy/rpc_user_config.yml file to configure networking. 1. Configure the IP address ranges associated with each network in the cidr_networks section: 27

33 cidr_networks: # Container management network container: CONTAINER_MGMT_CIDR # Tunnel network tunnel: TUNNEL_CIDR # Storage network (optional) storage: STORAGE_CIDR Replace *_CIDR with the appropriate IP address range in CIDR notation. For example, /24. Note Use the same IP address ranges as the underlying physical network interfaces or bridges configured in Section 4.5, Configuring the network [18]. For example, if the container network uses /24, the CONTAINER_MGMT_CIDR should also use /24. The default configuration includes the optional storage and service networks. To remove one or both of them, comment out the appropriate network name. 2. Configure the existing IP addresses in the used_ips section: used_ips: - EXISTING_IP_ADDRESSES Replace EXISTING_IP_ADDRESSES with a list of existing IP addresses in the ranges defined in the previous step. This list should include all IP addresses manually configured on target hosts in the Section 4.5, Configuring the network [18], internal load balancers, service network bridge, and any other devices to avoid conflicts during the automatic IP address generation process. Note Add individual IP addresses on separate lines. For example, to prevent use of and 201: used_ips: Add a range of IP addresses using a comma. For example, to prevent use of : used_ips: , Configure load balancing in the global_overrides section: 28

34 global_overrides: # Internal load balancer VIP address internal_lb_vip_address: INTERNAL_LB_VIP_ADDRESS # External (DMZ) load balancer VIP address external_lb_vip_address: EXTERNAL_LB_VIP_ADDRESS # Load balancer hostname lb_name: LB_HOSTNAME # Container network bridge device management_bridge: "MGMT_BRIDGE" # Tunnel network bridge device tunnel_bridge: "TUNNEL_BRIDGE" Replace INTERNAL_LB_VIP_ADDRESS with the internal IP address of the load balancer. Infrastructure and OpenStack services use this IP address for internal communication. Replace EXTERNAL_LB_VIP_ADDRESS with the external, public, or DMZ IP address of the load balancer. Users primarily use this IP address for external API and web interfaces access. Replace LB_HOSTNAME with the hostname of the load balancer that resolves to the external, public, or DMZ IP address of the load balancer. Replace MGMT_BRIDGE with the container bridge device name, typically br-mgmt. Replace TUNNEL_BRIDGE with the tunnel/overlay bridge device name, typically brvxlan. 4. Configure optional networks in the provider_networks subsection: provider_networks: - network: group_binds: - glance_api - cinder_api - cinder_volume - nova_compute type: "raw" container_bridge: "br-storage" container_interface: "eth2" ip_from_q: "storage" Note The default configuration includes the optional storage and service networks. To remove one or both of them, comment out the entire associated stanza beginning with the - network: line. 5. Configure OpenStack Networking tunnel/overlay network in the provider_networks subsection: 29

35 provider_networks: - network: group_binds: - neutron_linuxbridge_agent container_bridge: "br-vxlan" container_interface: "eth10" ip_from_q: "tunnel" type: "vxlan" range: "TUNNEL_ID_RANGE" net_name: "vxlan" Replace TUNNEL_ID_RANGE with the tunnel ID range. For example, 1: Configure OpenStack Networking provider networks in the provider_networks subsection: provider_networks: - network: group_binds: - neutron_linuxbridge_agent container_bridge: "br-vlan" container_interface: "eth11" type: "flat" net_name: "vlan" - network: group_binds: - neutron_linuxbridge_agent container_bridge: "br-vlan" container_interface: "eth11" type: "vlan" range: VLAN_ID_RANGE net_name: "vlan" Replace VLAN_ID_RANGE with the VLAN ID range for each VLAN provider network. For example, 1:1000. Create a similar stanza for each additional provider network Configuring target hosts Modify the /etc/rpc_deploy/rpc_user_config.yml file to configure the target hosts. Warning Do not assign the same IP address to different target hostnames. Unexpected results may occur. Each IP address and hostname must be a matching pair. To use the same host in multiple roles, for example infrastructure and networking, specify the same hostname and IP in each section. Use short hostnames rather than fully-qualified domain names (FQDN) to prevent length limitation issues with LXC and SSH. For example, a suitable short hostname for a compute host might be: Compute Configure a list containing at least three infrastructure target hosts in the infra_hosts section: 30

36 infra_hosts: infra01: ip: INFRA01_IP_ADDRESS infra02: ip: INFRA02_IP_ADDRESS infra03: ip: INFRA03_IP_ADDRESS infra04:... Replace *_IP_ADDRESS with the IP address of the br-mgmt container management bridge on each infrastructure target host. Use the same netblock as bond0 on the nodes, for example: infra_hosts: infra01: ip: infra02: ip: infra03: ip: (Optional) Add container affinity to infrastructure hosts. By default, the installation process deploys one instance of each container type per host. Affinity enables deployment of different quantities of each container type per host. For example, deploying two Memcached containers on an infrastructure host: infra_hosts: infra01: affinity: memcached_container: 2 Affinity also supports deploying zero instances of a container type per host. For example, deploying zero Galera containers on an infrastructure host: infra_hosts: infra01: affinity: galera_container: 0 For more information, see the list of container types that support affinity [32]. 3. Configure a list containing at least one network target host in the network_hosts section: network_hosts: network01: ip: NETWORK01_IP_ADDRESS network02:... Replace *_IP_ADDRESS with the IP address of the br-mgmt container management bridge on each network target host. 4. Configure a list containing at least one compute target host in the compute_hosts section: 31

37 compute_hosts: compute001: ip: COMPUTE001_IP_ADDRESS compute002:... Replace *_IP_ADDRESS with the IP address of the br-mgmt container management bridge on each compute target host. 5. Configure a list containing at least one logging target host in the log_hosts section: log_hosts: logging01: ip: LOGGER1_IP_ADDRESS logging02:... Replace *_IP_ADDRESS with the IP address of the br-mgmt container management bridge on each logging target host. 6. Configure a list containing at least one optional storage host in the storage_hosts section: storage_hosts: storage01: ip: STORAGE01_IP_ADDRESS storage02:... Replace *_IP_ADDRESS with the IP address of the br-mgmt container management bridge on each storage target host. Each storage host also requires additional configuration to define the backend driver. Note The default configuration includes an optional storage host. To install without storage hosts, comment out the stanza beginning with the storage_hosts: line. The following container types support affinity in typical deployments: cinder_api_container galera_container glance_container glance_container heat_apis_container heat_engine_container horizon_container memcached_container nova_api_ec2_container nova_api_metadata_container 32

38 nova_api_os_compute_container nova_cert_container nova_conductor_container nova_scheduler_container nova_spice_console_container 5.4. Configuring service passwords Change the default password for all services in the /etc/rpc_deploy/ user_variables.yml file. Consider using Ansible Vault to increase security by encrypting this file. Note that the following options configure passwords for the web interfaces: keystone_auth_admin_password configures the admin tenant password for both the OpenStack API and dashboard access. kibana_password configures the password for Kibana web interface access. The openstack-ansible repository provides a script to generate random passwords for each service. For example: # cd /opt/openstack-ansible/scripts # pw-token-gen.py --file /etc/rpc_deploy/user_variables.yml To regenerate existing passwords, add the --regen flag Configuring the hypervisor (optional) By default, the KVM hypervisor is used. If you are deploying to a host that does not support KVM hardware acceleration extensions, select a suitable hypervisor type such as qemu or lxc. To change the hypervisor type, uncomment and edit the following line in the /etc/ rpc_deploy/user_variables.yml file: # nova_virt_type: kvm 5.6. Configuring the Image Service (optional) In an all-in-one deployment with a single infrastructure node, the Image Service uses the local file system on the target host to store images. In a Rackspace deployment with three infrastructure nodes, the Image Service must use Cloud Files or NetApp. The following procedure describes how to modify the /etc/rpc_deploy/user_variables.yml file to enable Cloud Files usage. 1. Change the default store to use swift, the underlying architecture of Cloud Files: glance_default_store: swift 33

39 2. Set the appropriate authentication URL: For US Rackspace cloud accounts: rackspace_cloud_auth_url: For UK Rackspace cloud accounts: rackspace_cloud_auth_url: 3. Set Rackspace cloud account credentials by locating the RAX_CLOUD_TENANT_ID: a. Log into mycloud.rackspace.com as the desired user. b. Locate and click on the Account: $USERNAME link in the upper right section of the screen. c. Copy the Account Number shown. 4. Set the remaining Rackspace cloud account credentials with the RAX_CLOUD_TENANT_ID: rackspace_cloud_tenant_id: RAX_CLOUD_TENANT_ID rackspace_cloud_username: RAX_CLOUD_USER_NAME rackspace_cloud_password: RAX_CLOUD_PASSWORD 5. Change the glance_swift_store_endpoint_type from the default internalurl settings to public if needed. Glance services are typically backed by Rackspace cloud files in the Rackspace Data Centre. If the OpenStack environment needs to run outside the datacentre, adjust the key value: glance_swift_store_endpoint_type: publicurl 6. Replace RAX_CLOUD_* with the appropriate Rackspace cloud account credential components. 7. Define the store name: glance_swift_store_container: STORE_NAME Replace STORE_NAME with the store name in Cloud Files. If the store doesn't exist, a new store will be created. 8. Define the store region: glance_swift_store_region: STORE_REGION Replace STORE_REGION with one of the following region codes: DFW, HKG, IAD, LON, ORD, SYD. Note UK Rackspace cloud accounts must use the LON region. US Rackspace cloud accounts can use any region except LON. 34

40 5.7. Configuring the Block Storage service (optional) By default, the Block Storage service uses the LVM backend. To use a NetApp storage appliance backend, edit the /etc/rpc_deploy/rpc_user_config.yml file and configure each storage node that will use it: Note Ensure that the NAS Team enables httpd.admin.access. 1. Add the netapp stanza under the cinder_backends stanza for each storage node: cinder_backends: netapp: The options in subsequent steps fit under the netapp stanza. Note The backend name is arbitrary and becomes a volume type within the Block Storage service. 2. Configure the storage family: netapp_storage_family: STORAGE_FAMILY Replace STORAGE_FAMILY with ontap_7mode for Data ONTAP operating in 7-mode or ontap_cluster for Data ONTAP operating as a cluster. 3. Configure the storage protocol: netapp_storage_protocol: STORAGE_PROTOCOL Replace STORAGE_PROTOCOL with iscsi for iscsi or nfs for NFS. For the NFS protocol, you must also specify the location of the configuration file that lists the shares available to the Block Storage service: nfs_shares_config: SHARE_CONFIG Replace SHARE_CONFIG with the location of the share configuration file. For example, /etc/cinder/nfs_shares. 4. Configure the server: netapp_server_hostname: SERVER_HOSTNAME Replace SERVER_HOSTNAME with the hostnames for both netapp controllers. 5. Configure the server API port: netapp_server_port: PORT_NUMBER Replace PORT_NUMBER with 80 for HTTP or 443 for HTTPS. 35

41 6. Configure the server credentials: netapp_login: USER_NAME netapp_password: PASSWORD Replace USER_NAME and PASSWORD with the appropriate values. 7. Select the NetApp driver: volume_driver: cinder.volume.drivers.netapp.common.netappdriver 8. Configure the volume backend name: volume_backend_name: BACKEND_NAME Replace BACKEND_NAME with a suitable value that provides a hint for the Block Storage scheduler. For example, NETAPP_iSCSI. 9. Check that the rpc_user_config.yml configuration is accurate: storage_hosts: xxxxxx-infra01: ip: container_vars: cinder_backends: limit_container_types: cinder_volume netapp: netapp_storage_family: ontap_7mode netapp_storage_protocol: nfs netapp_server_hostname: netapp_server_port: 80 netapp_login: rpc_cinder netapp_password: password volume_driver: cinder.volume.drivers.netapp.common.netappdriver volume_backend_name: NETAPP_NFS For netapp_server_hostname, specifiy the IP address of the Data ONTAP server. Include iscsi or NFS for the netapp_storage_family depending on the configuration. Add 80 if using HTTP or 443 if using HTTPS for netapp_server_port. The cinder-volume.yml playbook will automatically install the nfs-common file across the hosts, transitioning from an LVM to a NetApp backend Configuring Block Storage Service for multiple NetApp backends (optional) The Block Storage service supports multiple backends. To use multiple NetApp backends, edit the /etc/rpc_deploy/rpc_user_config.yml file and configure each storage node to the designated NetApp backend: 1. For each NetApp backend, add a NetApp stanza name under the cinder_backends stanza for the storage node. For example: storage_hosts: xxxxxx-infra01: 36

42 ip: container_vars: cinder_backends: limit_container_types: cinder_volume netapp1:... xxxxxx-infra02: ip: container_vars: cinder_backends: limit_container_types: cinder_volume netapp2: For each netapp backend, set the options as described in the Section 5.7, Configuring the Block Storage service (optional) [35] procedure, steps Check that the rpc_user_config.yml configuration is accurate, for example: storage_hosts: xxxxxx-infra01: ip: container_vars: cinder_backends: limit_container_types: cinder_volume netapp1: netapp_storage_family: ontap_7mode netapp_storage_protocol: nfs netapp_server_hostname: netapp_server_port: 80 netapp_login: rpc_cinder netapp_password: password volume_driver: cinder.volume.drivers.netapp.common.netappdriver volume_backend_name: NETAPP_NFS xxxxxx-infra02: ip: container_vars: cinder_backends: limit_container_types: cinder_volume netapp2: netapp_storage_family: ontap_7mode netapp_storage_protocol: nfs netapp_server_hostname: netapp_server_port: 80 netapp_login: rpc_cinder netapp_password: password volume_driver: cinder.volume.drivers.netapp.common.netappdriver volume_backend_name: NETAPP_NFS 5.8. Configure the Block Storage Service with NFS protocols (optional). If the NetApp backend is configured to use an NFS storage protocol, edit /etc/ rpc_deploy/rpc_user_config.yml, and configure the NFS client on each storage node that will use it. 37

43 1. Add the nfs_client stanza under the container_vars stanza for each storage node: container_vars: nfs_client: 2. Configure the location of the file that lists shares available to the block storage service. This configuration file must include nfs_shares_config: nfs_shares_config: SHARE_CONFIG Replace SHARE_CONFIG with the location of the share configuration file. For example, /etc/cinder/nfs_shares. 3. Configure one or more NFS shares: shares: - { ip: NFS_HOST, share: NFS_SHARE } Replace NFS_HOST with the IP address or hostname of the NFS server, and the NFS_SHARE with the absolute path to an existing and accessible NFS share Creating Block Storage availability zones (optional) Multiple availability zones can be created to manage Block Storage storage hosts. Edit the /etc/rpc_deploy/rpc_user-config.yml file to set up availability zones. 1. For each cinder storage host, configure the availability zone under the container_vars stanza: cinder_storage_availability_zone: CINDERAZ Replace CINDERAZ with a suitable name. For example cinderaz_2 2. If more than one availability zone is created, configure the default availability zone for scheduling volume creation: cinder_default_availability_zone: CINDERAZ_DEFAULT Replace CINDERAZ_DEFAULT with a suitable name. For example, cinderaz_1. The default availability zone should be the same for all Cinder storage hosts. Note If the cinder_default_availability_zone is not defined, the default variable will be used. 38

44 6. Foundation playbooks Figure 6.1. Installation workflow The main Ansible foundation playbook prepares the target hosts for infrastructure and OpenStack services and performs the following operations: Perform deployment host initial setup Build containers on target hosts Restart containers on target hosts Install common components into containers on target hosts 6.1. Running the foundation playbook 1. Change to the /opt/openstack-ansible/rpc_deployment directory. 2. Run the host setup playbook, which runs a series of sub-playbooks: $ ansible-playbook \ playbooks/setup/host-setup.yml Confirm satisfactory completion with zero items unreachable or failed: PLAY RECAP ********************************************************************... deployment_host : ok=18 changed=11 unreachable=0 failed= Troubleshooting Q: How do I resolve the following error after running a playbook? failed: [target_host] => (item=target_host_horizon_container-69099e06) => {"err": "lxc-attach: No such file or directory - failed to open 39

45 '/proc/12440/ns/mnt'\nlxc-attach: failed to enter the namespace\n", "failed": true, "item": "target_host_horizon_container-69099e06", "rc": 1} msg: Failed executing lxc-attach. A: The lxc-attach sometimes fails to execute properly. This issue can be resolved by running the playbook again. 40

46 7. Infrastructure playbooks Figure 7.1. Installation workflow The main Ansible infrastructure playbook installs infrastructure services and performs the following operations: Install Memcached Install Galera Install RabbitMQ Install Rsyslog Install Elasticsearch Install Logstash Install Kibana Install Elasticsearch command-line utilities Configure Rsyslog 7.1. Running the infrastructure playbook 1. Change to the /opt/openstack-ansible/rpc_deployment directory. 2. Run the infrastructure setup playbook, which runs a series of sub-playbooks: $ ansible-playbook \ playbooks/infrastructure/infrastructure-setup.yml Confirm satisfactory completion with zero items unreachable or failed: PLAY RECAP ********************************************************************... 41

47 deployment_host : ok=27 changed=0 unreachable=0 failed= Verifying infrastructure operation Verify the database cluster and Kibana web interface operation. Procedure 7.1. Verifying the database cluster 1. Determine the Galera container name: $ lxc-ls grep galera infra1_galera_container-4ed0d84a 2. Access the Galera container: $ lxc-attach -n infra1_galera_container-4ed0d84a 3. Run the MariaDB client, show cluster status, and exit the client: $ mysql -u root -p MariaDB> show status like 'wsrep_cluster%'; Variable_name Value wsrep_cluster_conf_id 3 wsrep_cluster_size 3 wsrep_cluster_state_uuid bbe3f0f6-3a88-11e4-bd8f-f7c9e138dd07 wsrep_cluster_status Primary MariaDB> exit The wsrep_cluster_size field should indicate the number of nodes in the cluster and the wsrep_cluster_status field should indicate primary. Procedure 7.2. Verifying the Kibana web interface 1. With a web browser, access the Kibana web interface using the external load balancer IP address defined by the external_lb_vip_address option in the /etc/ rpc_deploy/rpc_user_config.yml file. The Kibana web interface uses HTTPS on port Authenticate using the username kibana and password defined by the kibana_password option in the /etc/rpc_deploy/user_variables.yml file. 42

48 8. OpenStack playbooks Figure 8.1. Installation workflow The main Ansible OpenStack playbook installs OpenStack services and performs the following operations: Install common components Create utility container that provides utilities to interact with services in other containers Install Identity (keystone) Generate service IDs for all services Install the Image Service (glance) Install Orchestration (heat) Install Compute (nova) Install Networking (neutron) Install Block Storage (cinder) Install Dashboard (horizon) Reconfigure Rsyslog 8.1. Utility Container Overview The utility container provides a space where miscellaneous tools and other software can be installed. Tools and objects can be placed in a utility container if they do not require a dedicated container or if it is impractical to create a new container for a single tool or object. Utility containers can also be used when tools cannot be installed directly onto a host. For example, the tempest playbooks are installed on the utility container since tempest testing does not need a container of its own. For another example of using the utility container, see Section 8.3, Verifying OpenStack operation [45]. 43