1 How Liferay Is Improving Quality Using Hundreds of Jenkins Servers James Min Sr. Consultant, Liferay, Inc. liferay.com
2 Liferay Background Open Source Portal & Collaboration Java Platform 250 employees worldwide In its 11th year of development, Liferay Portal is today s most popular and widely downloaded open source portal platform Over 4 Million downloads and 50,000 downloads per month 46,034 registered users on Liferay.com 15,758 forum participants 161,641 forum posts Over 50 active volunteer contributors
3 Challenges at Liferay
4 Liferay Portal Needs CI For: Liferay Portal Core Liferay Portal Plugins Deployment platforms and environments (provisioning)
5 Before Jenkins Manual Setup of Environments Manual Testing As Liferay supports more platforms, the effort to keep up increases exponentially!
6 Liferay s Early Jenkins Setup 100 unclustered servers Each physical machine set up to run up to 4 VMs with 13 VMs (vmdk files) loaded onto an extra hard disk Hardware Specs: INTEL CPU C2Q Q9550s 2.83ghz INTEL MB G41MJ MITX LGA775 Intel X25 M Mainstream SATA Solid State Drive 80GB 2.5" 9.5mm SSD
7 Jenkins Manages Tasks/Tests Deploy every permutation of application server/database/operating system Run smoke tests If smoke test fails, Jenkins automatically bypasses executing any of the subsequent downstream jobs related to that environment. If smoke test passes, run tests (Selenium) downstream
8 Liferay Smoke Tests builds the portal war zips it up deploys it into an app server starts the app server logs into the portal asserts that it was able to log in logs out asserts log out shuts down the app server
9 Liferay Smoke Tests If all the smoke test pass (e.g. something wrong): Test every deployment w/ all tests (10,000) it takes about 7 10 hours after the initial build
10 Growing Pains Liferay Core Development: Scotty, We need more power! Liferay QA: I m givin her all she s got, Captain!
11 Growing Pains As Liferay supports more platforms and features, testing takes longer % utilization on each machine Jenkins, unclustered, has no master to manage, and creates inefficiency (i.e. idle servers after testing is done) As Liferay s core code base and the number of Liferay plugins grow, there is more to test.
12 Liferay s Evolving Setup 100 isolated Jenkins nodes = Bottlenecks & Inefficency When one node is finished it does not necessarily start another task Other tests do not start until they are manually started or the last one is finished
13 Liferay s Evolving Setup Jenkins Cluster with master and 250 slaves Continuous deploying of new VMs and building new environments while other nodes finish tasks/tests.
14 Liferay s Current Setup 250 physical servers at colo facility Jenkins 1.4 Cluster Selenium Grid 10,000 tests per version
15 Liferay s Future Plans 500 Servers by end of Servers by mid 2012 Expansion room for 2000 total servers Move from a colo to in house managed facility
16 Liferay s Future Plans & Challenges How to deploy Jenkins en masse? How to deploy the configuration and plugins en masse? Currently have a homegrown framework where we utilize batch scripting and a NAS that's mapped to all of the servers, and remotely shutdown/restart/format/reset the servers and/or Jenkins instances
17 Liferay s Future Plans & Challenges Jenkins calls test scripts in Liferay that handle launching a Selenium server instance alongside a Liferay instance for testing Jenkins Selenium Grid Cluster!!! Remote configure Jenkins servers en masse as well using ant/batch scripting so we can bring down a server and repurpose it pretty quickly through utilizing the scripting framework Unexplored territory for Liferay Looking forward to Ryan Campbell & Kohsuke Kawaguchi s talk later
18 Why Our Own Cloud vs. Cloud Service? Should we have used a cloud service setup (PaaS)? For Liferay, we found if you know your utilization, building our own was more cost effective We calculated it to cost roughly 10 12x more to do it on the cloud than to house it internally even when you factor in all the other costs such as maintenance, electricity, bandwidth, etc. Cloud service is very useful when you don't know the demand or if is unpredictable. If you don't know whether you need 10 or 100 or 1000 servers, and you don't know how often they are actually being used, then a cloud service is cheaper because you have lower up front costs. For Liferay, the cloud service is definitely more expensive per unit cost (i.e. cloud vendors have to make money some how). Since we know our usage (close to 90% because they're build servers), it's a LOT cheaper for us to do it in house.
19 Our Own Cloud vs. EC2? When we calculated for just 100 servers, based on our calculations, since we run 4 vms on each machine and we have utilization of at least 50%, that would be 300k in fees to Amazon a year. If we ran at 100%, we'd pay 600k a year to Amazon So for 100k for hardware (one time expense), plus 50k in colocation fees a year, plus X $ for salary of maintaining the servers per year, it's still cheaper than Amazon. The reason it's cheap for us is because we actually USE it a lot and we have decent expertise.
20 Our Own Cloud vs. EC2? Most people use it a fraction % of the day, because their tests finish relatively quickly. Our tests are hours at a time. For Liferay, the cloud service is definitely more expensive per unit. Since we know our usage (close to 90% because they're build servers), it's a huge savings for us to do it in house.
21 Our Own Cloud vs. EC2? With $1 million worth of servers, (one time cost), for the same power, we d be paying Amazon $6 million in fees a year if we used it 100%. And the next year, we don't have to pay for the servers again! Just what's broken. Today, our internet fees at the colo (for 250 servers) is 96k a year. Before, for 100 servers, it was 50k a year. So to lower that, we're even going to move our colo inhouse (i.e. we're building our own colo now)
22 PaaS Cloud Is Still A Great Option UNLESS, they happen to be in a situation similar to Liferay. Have a huge need and a predictable utilization requirement Have in house expertise Most flexible option Weigh the one time capitalization cost vs. yearly operating costs
23 Thank You To Our Sponsors Platinum Sponsor Gold Sponsor Silver Sponsor Bronze Sponsors Coming Soon: The CloudBees Newsletter for Jenkins Please complete the Jenkins survey to help us better serve the community (bonus: a chance to win an Apple TV!)
24 Questions? James Min, Sr. Consultant / Sales Engineer
Many of our enterprise customers wanted dedicated virtual servers that offered a much higher degree of isolation... we needed to step up our virtualization efforts to stay competitive." Joe Young, Senior
Estimating the Cost of a GIS in the Amazon Cloud An Esri White Paper August 2012 Copyright 2012 Esri All rights reserved. Printed in the United States of America. The information contained in this document
Recommended Backup Strategy for FileMaker Server 10 and 11 for Macintosh & Windows Updated September 2010 This document provides a single cohesive resource for managing and understanding the backup strategy
VoIP 101: An introduction to the basics of Voice over Internet Protocol How to guide Introduction You may have heard of VoIP that it s the future of telephone service, and that you can save a lot of money
Software and services are moving off corporate and organizational networks and into the cloud. The trends are clear and the pace is accelerating. But what exactly is the cloud? Can client-server software
The Total Cost of (Non) Ownership of a NoSQL Database Cloud Service Jinesh Varia and Jose Papo March 2012 (Please consult http://aws.amazon.com/whitepapers/ for the latest version of this paper) Page 1
Whitepaper The top 10 advantages of a Windows based PBX Why your next Phone System should be software based This whitepaper outlines the top 10 advantages of choosing a Windows based PBX as your next phone
Develop an intelligent disaster recovery solution with cloud technologies IBM experts share their insight on how cloud technologies can help restore IT operations more quickly, reliably and cost-effectively
Is Connectivity A Human Right? For almost ten years, Facebook has been on a mission to make the world more open and connected. For us, that means the entire world not just the richest, most developed countries.
Cloud Computing: A Brief Summary Lucid Communications Limited Prepared by Neil Turner September 2009 I. Contents 1. Introduction...1 2. Software as a Service (SaaS)...2 2.1. Disadvantages...2 2.2. Advantages...2
Whitepaper The Top 10 Advantages of 3CX Phone System Why your next phone system should be software based and by 3CX This whitepaper outlines the top 10 advantages of choosing 3CX Phone System, a Windows
Implementing An Issue Management Solution Issue management means a lot of things to a lot of people. A quick search on Google will invoke definitions, descriptions, tips, software tools, and organizations
Windows Server 2008 R2 Hyper-V Live Migration Table of Contents Overview of Windows Server 2008 R2 Hyper-V Features... 3 Dynamic VM storage... 3 Enhanced Processor Support... 3 Enhanced Networking Support...
white paper Public or Private Cloud: The Choice is Yours Current Cloudy Situation Facing Businesses There is no debate that most businesses are adopting cloud services at a rapid pace. In fact, a recent
Whitepaper The Top 10 Advantages of a Windows Based PBX Why Your Next Phone System Should be Software Based This whitepaper outlines the top 10 advantages of choosing a Windows based PBX as your next phone
Red Hat Cloud Foundations: Cloud 101 Table of contents 2 Executive summary 3 Introduction 3 Clouds today and tomorrow 5 The cloud taxonomy 7 What cloud computing isn t 7 Why cloud computing 8 Who creates
Subtitle Business Phone Trends 2015 The Relentless March of Technology Business Phone Trends - 2015 Compare Business Products 2015 1 Contents The Trends That Will Define 2015...3 IPv6...3 The Cloud Will
QLIKVIEW AND THE CLOUD A QlikView Technology White Paper July 2012 qlikview.com Table of Contents Overview 4 Who Is This Paper For? 4 Cloud Terminology and QlikView 5 Software as a Service - SaaS 6 QlikView
ON-PREMISE VS. CLOUD-BASED SOLUTIONS Which is Best? A Dilemma for SMBs As the price of storage and bandwidth continues to drop fast, Cloudbased services are becoming more and more attractive to small and
Will Cloud Computing Work for the Life Sciences Industry? Even in a regulated industry there are many opportunities to start seeding the Clouds Cloud computing refers to the idea of delivering information
www.cloudtp.com Building Blocks of the Private Cloud Private clouds are exactly what they sound like. Your own instance of SaaS, PaaS, or IaaS that exists in your own data center, all tucked away, protected
Discovery and Usage data for Software License Management Is Microsoft SCCM the best solution? Many organizations use Microsoft s SCCM product as their primary software discovery method. Because SCCM is
Three steps to put Predictive Analytics to Work The most powerful examples of analytic success use Decision Management to deploy analytic insight in day to day operations helping organizations make more
VOXOX 5BENEFITS OF A HOSTED VoIP SOLUTION FOR MULTI-OFFICE BUSINESSES a VOXOX ebook 0Taking 2013 VOXOX, Inc. Communications to the Cloud: CONTENTS 2 INTRODUCTION 3 TOP CHALLENGES 7 FINDING A SOLUTION 12
A P P L I C A T I O N S A WHITE PAPER SERIES IMPLEMENTING A CLOUD COMPUTING MODEL IS A MAJOR CHALLENGE. ENTERPRISES NEED TO TAKE A HOLISTIC APPROACH TO ENSURE ACCESSIBILITY, FLEXIBILITY AND SECURITY. Journey