Containers with Google: from Borg to Kubernetes

Containers with Google: from Borg to Kubernetes Jens Bussmann Cloud Platform Lead, NACE Google Filip Grzadkowski Warsaw Kubernetes Team Lead Google

Building what s next 3 We set out to build a better cloud for us. Now, we re giving it to you. 3

Building what s next 4 4

Google Innovations in Software Kubernetes FlumeJava Borg Spanner Colossus Bigtable Dremel 2015 MapReduce 2012 GFS 2010 2006 2008 2004 2002 5

Containers At Google, everything runs in a container

This is vanilla virtualization Guest environment app code libraries guest kernel Hypervisor 10

It has downsides: Not portable & opaque Guest environment app code libraries machine image locked into a platform guest kernel Hypervisor 11

It has downsides: No Isolation Guest environment app code app code??? dependency libraries guest kernel Hypervisor 12

It has downsides: Little Reuse redundant Guest environment Guest environment Guest environment app code app code app code libraries libraries libraries guest kernel guest kernel guest kernel Hypervisor 13

Containers create a better abstraction layer Guest environment app code cut here libraries guest kernel Hypervisor 14

Containers at Google Google is using container technology for more than 10 years Everything at Google runs in a container 15

Containers at Google Google is using container technology for more than 10 years Everything at Google runs in a container We launch over 2 billion new containers per week 16

Containers Google and containers Google and container technology Released CGroups Limited isolation 2004 Kubernetes Released LMCTFY 2006 2013 2014 Confidential & Proprietary 17

Containers Google and containers Google and container technology Released CGroups Limited isolation Kubernetes Released LMCTFY 2004 2006 Borg 2013 2014 10+ years of experience Confidential & Proprietary 18

Kubernetes Container orchestrator Builds on containers (Docker, Rocket) Handles container and node failure Multiple cloud and bare-metal environments 100% Open Source, written in Go Let users manage applications, not machines 19

Open Source 5k Commits in 1.2 +50% Unique Contributors Top 0.01% of all Github Projects 1200+ External Projects Based on K8s Companies Contributing Companies Using 20

Kubernetes Components Confidential & Proprietary 21

Pods A group of one or more containers sharing fate Containers in a pod run in a shared context (tcp/ip, sockets) Can communicate via localhost Considered to be relatively ephemeral (rather than durable) entities Pod Container 1 Container 2 22

Pod example apiversion: v1 kind: Pod metadata: name: wordpress spec: containers: - image: wordpress name: wordpress env: - name: WORDPRESS_USER value: wp_user Public image from Docker Hub. Private images have the form gcr.io/<your-project>/<your-image> 23

Nodes Pod 1 A worker machine in Kubernetes that will host one or many pods May be a VM or physical machine, depending on the cluster Restarted automatically after failure on GCE Pod 2 Node 1 24

Services Pod 1 Your load balancers, so to speak A stable endpoint that hides the implementation behind (pods or external) All pods will see your services as env variables or DNS entries Different Types: ClusterIP NodePort LoadBalancer Pod 2 Node 1 wordpress-service 146.148.3.114:80 Pod 3 Pod 4 Node 2 25

Labels type=wordpress Pod 1 Any combination of key-value pairs Pod 2 data-center=east release=production version=1.2 tier=front-end Can be applied to pods, services, nodes Node 1 wordpress-service 146.148.3.114:80 Pod 3 Pod 4 Node 2 26

Service example apiversion: v1 kind: Service metadata: name: wpfrontend spec: type: LoadBalancer ports: - port: 80 targetport: 80 protocol: TCP selector: name: wordpress Type of service: ClusterIP, NodePort, LoadBalancer Label to select the pods that will receive traffic 27

Replica Sets (since 1.2) Pod 1 Ensures a specified number of pod replicas are running with the specified labels Will react if a pod dies unexpectedly Can autoscale Pod 2 Node 1 wordpress-rc replicas: 4 Pod 3 Pod 4 Node 2 28

Replica Sets (1.2) vs Replication Controller (1.1) Replica Sets will replace Replication Controller The only difference is more flexible label selector support Not to be used directly. Use Deployment instead # equality-based environment = production tier!= frontend # set-based environment in (production, qa) tier notin (frontend, backend) partition!partition 29

Deployments (since 1.2) Provides declarative updates for Pods and Replica Sets Describe the desired state in a Deployment object, and the Deployment controller will get there at a controlled rate Pause, Resume, Rollback deployments Deployments keep a history 30

Deployment example apiversion: extensions/v1beta1 kind: Deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerport: 80 if too few, start new replicas if too many, kill some 31

Deployments $ kubectl create -f docs/user-guide/nginx-deployment.yaml deployment "nginx-deployment" created $ kubectl get deployments NAME DESIRED nginx-deployment 3 CURRENT 0 UP-TO-DATE 0 AVAILABLE 0 AGE 1s $ kubectl get deployments NAME DESIRED nginx-deployment 3 CURRENT 3 UP-TO-DATE 3 AVAILABLE 3 AGE 18s 32

Update a Deployment apiversion: extensions/v1beta1 kind: Deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerport: 80 apiversion: extensions/v1beta1 kind: Deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.9.1 ports: - containerport: 80 33

Update a Deployment $ kubectl apply -f docs/user-guide/new-nginx-deployment.yaml deployment "nginx-deployment" configured $ kubectl get deployments NAME DESIRED nginx-deployment 3 CURRENT 3 UP-TO-DATE 0 AVAILABLE 3 AGE 20s $ kubectl get deployments NAME DESIRED nginx-deployment 3 CURRENT 3 UP-TO-DATE 3 AVAILABLE 3 AGE 36s 34

Resources: Requests / Limit Set up required and maximum resources for the pod The pod can only be scheduled in a node with enough capacity for all its containers spec: containers: - name: db image: mysql resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" If a pod requests cpu: 0 it can always be scheduled 35

Ranking potential nodes Prefer node with most free resource left after the pod is deployed Minimise number of Pods from the same service on the same node CPU and memory are balanced after the Pod is deployed Node Affinity (since 1.2): use label filters to alter node selection 36

Other components Persistent Volume: attach a persistent disk to a pod. Secrets: inject sensitive information into your pods, reducing the risk of accidental exposure Namespaces: scope for object names. Names need to be unique within a namespace. Daemon Set: All nodes run a replica of a pod Jobs: creates pods and ensures that a specified number of them successfully terminate. 37

Host it in the cloud or on premise Confidential & Proprietary 38

Google Container Engine The easiest way of getting started with Kubernetes Managed by Google Cloud Logging and Monitoring integrated out-of-the-box Once connected, it s just Kubernetes # Create cluster gcloud container clusters create hello-world --num-nodes 3 # Connect Kubernetes gcloud container clusters get-credentials hello-world # Use Kubernetes kubectl get pods 39

(Coming soon) A collaboration between Google and Red Hat Deploy your Docker containers in a IaaS platform managed by Red Hat Migrate your application seamlessly between Google Cloud and your own premises Portable: move your application to your own Data Center using Open Shift Online or Open Shift Enterprise. Confidential & Proprietary 40

Hosting Kubernetes Hosting for Kubernetes in the hybrid cloud Online, Enterprise or Dedicated Managed by Red Hat or hosted yourself Hosting for Kubernetes in Managed by Google Scale with a click Confidential & Proprietary 41

Build What s Next