OpenStack Use Case at GREE

Transcription

1 OpenStack Use Case at GREE GREE, Inc. 2014/2/14 OpenStack Days Tokyo 2014

2 Self Introduction Koichi Watanabe (29) Infrastructure Dept., GREE Infrastructure Design and Development Favorites: 富士山, 富士宮, 日馬富士.. sth like Fuji Yohei Matsuhashi (27) Infrastructure Dept., GREE Infrastructure Design and Development Favorites: ものづくり (Robotics, etc)

3 Agenda Introduction Infrastructure Overview (before OpenStack) Why Virtualization? System Overview Implementation Issues from testing Issues from operation Recent Work Conclusion

4 Introduction About Us Company GREE, Inc. Est. Location Dec 7th, 2004 Roppongi, Tokyo Business Social Gaming Business Social Media Business Platform Business Advertising & Ad Network Business Licensing & Merchandising Business Venture Capital Business

5 Introduction Social Games Many kinds of games 踊り子クリノッペ 釣り スタ 探検ドリランド 聖戦ケルベロス

6 Infrastructure Overview (before OpenStack)

7 Infrastructure Overview Over 1,250+ products

8 Infrastructure Overview GREE s Server Farm On-premise hundreds of servers per service/game Recurrent Peak traffic several times per day

9 Infrastructure Overview Server Dashboard Server Management Portal, with Auto-Configuration features

10 Infrastructure Overview Controlled by RESTful API { results => { [{ property => { server_type =>... bootstrap config deploy etc Web Portal Monitoring Alert Products Products LVSProducts LVS Server Dashboard RESTful API Tools Web Web Web DB DB DB CMDB

11 Infrastructure Overview Elasticity Able to scale-in/scale-out all server components Controlled with Server Dashboard LVS LVS LVS LVS Web Web Web Web Web DB Master DB Slave DB Slave DB Master DB Slave

12 Infrastructure Overview Server Resource Pool Activate and use servers from common resource pool as and when necessary Product A LVS Manage LVS Web Web Web DB Master DB Slave DB Slave Common Resource Pool Product B LVS Server Dashboard LVS BM BM BM Web Web Web BM BM BM DB Master DB Slave DB Slave BM BM BM

13 Infrastructure Overview Operation Cycle Spare servers are used to deal with failures Product LVS Server Dashboard LVS Deploy Web Web Web Common Resource Pool Web BM BM BM BM BM BM BM BM BM Service-out DB Master DB Slave DB Slave

14 Why Virtualization?

15 Why Virtualization? Use case Improve server resource usage we have various services and roles some of them could be run on smaller servers eg. gateway, batch servers Improve automation Automate machine and software tests boot, configure, test and destroy Reduce operation costs HA/FT Live migration Floating IP

16 Why Virtualization? Points to consider Reason Comments Reduced Performance Approx 5-50% of Hypervisor resource overhead Reduced Dependability Noisy Neighbour problem Decreased Visibility More complex architecture More Dependencies More external packages, such as OpenStack

17 Why Virtualization? Evaluation Summary Reason Comments Reduced administration effort Automation tools are mandatory Reduced fixed costs Reduced overall server resource usage Improved scalability Machine provisioning is faster than physical machines Increased flexibility Multi-tenancy, better network and resource management Increased application availability HyperVisor overhead is still a drawback Reduced development effort Test automation would be helpful

18 Why Virtualization? OpenStack x GREE Major evaluation axis OSS All API All Python Scalability Loose coupling in each elementals Multi-tenancy Compatible with Swift Familiar with our architecture Possible to use current servers Very active contributors and community Internet P V Common P P V P P B A P P Physical Virtual private GIA2- Server: 物理, Global global Network: 疎通, Private Network: 不通 P V Server Server network network GIA2v - Server: 物理 & 仮想, Global Network:疎通, Private Network: 不通

19 Overview of OpenStack Environment Current Production Environment (May 2013) Category Comments OpenStack Folsom Hypervisor K Host OS Ubuntu Networking Open vswitch Deployment Tool Chef Storage L

20 System Overview

21 System Overview OpenStack integration to our existing system Manage s and BMs on the same interface Sync info with Server Dashboard User Operat ion provisioning servers Server Dashboard System Info Product B Product A BM BM BM deploying contents Sync Delivery System Delivery BM BM BM Delivery

22 System Overview Rapid Scale In / Out On-Demand delivery Add s from common server pool in OpenStack Improved server deployment time Product LB Web LB Web Server Pool Web Add Delete DB DB DB

23 System Overview Hybrid BMs and s Able to choose BM or depending on app s workload Baremetal : high I/O : low resource usage Net IO bound Disk IO bound DB KVS Baremetal Web Stg LVS Dev Hypervisor

24 System Overview QoS Guaranteed s performances Network bandwidth Disk IOPS NIC Restrict bandwidth Restrict IOPS Local Storage 物理

25 System Overview Zero-downtime migration Migrate between BM using automation tools example : Product A Server Dashboard Web LVS 3. Bu ild & 2. Sync 4. switchover Web Depl oy Web 1. Add LVS Web

26 Implementation

27 Implementation GREE is completely built on OSS (not in production) and more!!

28 Implementation Deploying OpenStack Auto-provisioning with Chef Middleware Controller Compute Provisioning flow Install packages Performance tuning Apply patches etc. Deploy Compute Controller DB MQ

29 Implementation Middleware Component Middleware Redundant Database MySQL Keepalived Messaging queue RabbitMQ Pacemaker + DRBD Keepalived MySQL MySQL Pacemaker RabbitMQ RabbitMQ DRBD

30 Implementation OpenStack Controller OpenStack APIs Keystone :: Authentication Nova :: Compute Quantum/Neutron :: Network Glance :: Image Cinder :: Volume Redundant tool Keepalived

31 Implementation Compute Node Improve K performance virtio :: storage vhost :: network hugepages :: memory Able to apply QoS cgroup :: disk io tc :: traffic control How Chef configuration deployment OpenStack patching

32 Implementation Region / Availability Zone / Aggregate Concept Apply to Region Datacenter Availability Zone (AZ) Rack Aggregate (AG) Hardware type Region : DC-A AZ : rack-a-1 Region : DC-B AZ : rack-a-2 AG : hardware-x AG : hardware-y

33 Implementation Resource management during deployment Manage network IO by traffic control Prioritize disk IO with ionice Use local image cache in the hypervisor BM BM Traffic control Copy Cache 物理 Image

34 Implementation Original Operation Tools OpenStack command wrappers manage tenant/user/network include failsafe check mechanism customize s parameters placement scheduler Selects an appropriate place depending on the service and the server role tenant: tenanta network: vlan: 100 cidr: /24... create setup system Validator configuration yaml

35 Implementation Gathering 100+ RRD metrics Load measuring ex. memory usage, disk usage, assigned CPU cores

36 Implementation Managing OpenStack API Use reverse proxy in front of OpenStack APIs Apply Access Control List and SSL/TLS API load balancing Valid user access control Attacker Reverse Proxy load balancing

37 Issues from testing

38 Issues from testing 1/3 Connection temporary fails when ovs-agent is restarted Issue ovs flow entry is re-initialized upon restart Solution Base policy of not restarting agent Service out before restarting the agent 1. service out 3. service in Server Dashboard 2. restart quantum-ovs-agent Compute

39 Issues from testing 2/3 Invalid redundancy of DHCP servers Issue dhcp-agent couldn t failover leased IP addresses Depends on DHCP protocol specification Solution Use DHCP authoritative mode or Use static IP address authoritative mode dnsmasq dnsmasq lease network information

40 Issues from testing 3/3 Unable to set VIP for LVS Issue Denied VIP communication by security group Solution Modified security group rules Apply patches by chef VIP keepalived set iptables Controller security group modified

41 Issues from testing Extra Issues Issue Cause memory K suddenly fails overcommit Solution apply failsafe mechanism in the placement scheduler monitoring overcommit status and more!! [Caution] Some of issues we experienced while testing OpenStack folsom have already been fixed in the latest release Solutions we introduced may not be the best ways (as you may know )

42 Issues from operation

43 Issues from operation 1/3 DB record consumption by keystone token Issue Accumulating keystone tokens continuously Solution Set cron to delete expired tokens in database (Otherwise, migrate keystone backend to memcache) MySQL token-get insert access token token token token token token

44 Issues from operation 2/3 Taking measures to DDoS attack Issue OpenStack controller could have many global IPs ssh brute force attack, etc Solution Set filter rules by iptables (Otherwise, turn off DHCP server) Valid user Filter Attacker

45 Issues from operation 3/3 nova-compute process down after deleting flavor Issue Nova-compute process crash when a running flavor is deleted Solution Apply failsafe check mechanism by our tools delete flavor Validator

46 Issues from operation Extra Issues Issue Cause Solution s for DB Overhead of the K Use bare metal servers (Otherwise, replace to SSD/FusionIO) K bugs bugs in para-virt module. ex. kvm-clock apply workarounds and more!! [Caution] Some of issues we experienced while testing OpenStack folsom have already been fixed in the latest release Solutions we introduced may not be the best ways (as you may know )

47 Recent Work

48 Recent Work OpenStack x Vagrant for Automated Integration Test Tenant Web git repository LVS Spec DB OpenStack API Web-Server jenkins Data Center

49 Recent Work OpenStack x Vagrant for Automated Integration Test Automated Integration Test Hook Delivery s Configuration Deploy Codes Test ( Destroy) Create staging environments instantly with latest codes staging environment vagrant up launch Web: apache DB: MySQL Web: apache KVS: Flare LB: Nginx

50 Recent Work Step By Step Improve UI/UX and tools Adopt Domain Driven Development Networking Edge-to-Edge L2 Overlay Taking a look into OpenStack L3 Agent... Linux Container Docker Experiments in On-premise infrastructure Log-Manageable immutable infrastructure Blue-Green deployment

51 Conclusion

52 Conclusion Impressions of OpenStack Should design on own workload and requirements We have achieved High-Availability on application layer No SPoF in all systems including s application service Adopted local disk storage for s Test, test, and test... Understanding how it works Say No to extra features Many new projects on OpenStack Right person in the right place Some features worked only in devstack... Loose coupling in each components, but tight coupling as OpenStack cluster Upgrading OpenStack is a living hell

53 Conclusion Expectation for OpenStack Free to choose, free to design Respect for the culture as Open-Source Core infrastructure improvement More than PaaS or other as a Service Rolling Update HA/FT PCI Path Through etc.. Significant storage solution for Cinder

54