1 Using NixOS for declarative deployment and testing Sander van der Burg Eelco Dolstra Delft University of Technology, EEMCS, Department of Software Technology February 5, 2010
2 Linux distributions There are a wide range of Linux distributions available, each having different properties and goals.
3 Software deployment Software deployment All of the activities that make a software system available for use Carzaninga et al. Activities Install a Linux distribution with some desired packages Adapt/tweak configuration files Install custom pieces of software Upgrade a system
4 Deployment scenario Single installation
5 Deployment scenario Multiple installations Machines are connected and dependent on each other
6 Deployment scenario Virtual machines
7 Challenges Deploying a single machine is hard Takes some effort Upgrading may break the system Deploying a distributed environment is even harder Machines may be dependent on each other, e.g. web application using a database While upgrading, downtimes may occur Deploying (a network of) virtual machines is also hard Takes quite some effort to perform system integration tests
8 NixOS A GNU/Linux distribution using the Nix package manager
9 Nix store Main idea: store all packages in isolation from each other: /nix/store/rpdqxnilb0cg... -firefox Paths contain a 160-bit cryptographic hash of all inputs used to build the package: Sources Libraries Compilers Build scripts... /nix/store l9w6773m1msy...-openssh-4.6p1 bin ssh sbin sshd smkabrbibqv7...-openssl-0.9.8e lib libssl.so c6jbqm2mc0a7...-zlib lib libz.so im276akmsrhv...-glibc-2.5 lib libc.so.6
12 NixOS In NixOS, all packages including the Linux kernel and configuration files are managed by Nix. NixOS does not have directories such as: /lib and /usr NixOS has a minimal /bin and /etc But NixOS is more then just a distribution managed by Nix
14 NixOS configuration nixos-rebuild switch Nix package manager builds a complete system configuration Includes all packages and generates all configuration files, e.g. OpenSSH configuration Upgrades are (almost) atomic Components are stored safely next to each other, due to hashes No files are automatically removed or overwritten Users can switch to older generations of system configurations not garbage collected yet
15 NixOS bootloader
16 Distributed deployment NixOS has good properties for deployment of a single system Can we extend these properties to distributed systems?
17 Motivating example: Trac
18 Motivating example: Trac Trac can be deployed in a distributed environment: Subversion server Database server Web server
20 Distributed deployment nixos-deploy-network network.nix Build system configurations by the Nix package manager Transfer complete system and all dependencies to target machines in the network Efficient: only missing store paths must be transferred Safe: Existing configuration is not affected, because no files are overwritten or removed Activate new system configuration In case of a failure, roll back all configurations Relatively cheap operation, because old configuration is stored next to new configuration
21 Virtualization nixos-build-vms network.nix;./result/bin/nixos-run-vms Builds a network of QEMU-KVM virtual machines closely resembling the network of NixOS configurations We don t create disk images The VM mounts the Nix store of the host system using SMB/CIFS
22 Virtualization nixos-build-vms network.nix;./result/bin/nixos-run-vms Possible because complete configuration is in the Nix store This is efficient and safe due to the nature of the Nix store Components with same hash codes are shared between VMs The hash part of the name isolates components from each other Difficult to do for imperative Linux distributions, which have /etc, /usr, /lib directories.
26 Experience Distributed deployment of a Hydra build environment Continuous integration and testing of NixOS NixOS installer OpenSSH Trac NFS server Continuous integration and testing of various GNU projects Install NixOS system with bleeding edge glibc Other free software projects
27 Related work Examples: Cfengine Stork Related work uses convergent models NixOS models are congruent
28 Conclusion NixOS. A GNU/Linux distribution used to reliably deploy a complete system from a declarative specification nixos-deploy-network. Efficiently/Reliably deploy a network of NixOS machines nixos-build-vms. Efficiently generate a network of cheap NixOS virtual machines instances NixOS test driver. Perform distributed test cases in a network of NixOS virtual machines
29 References NixOS website: Nix. A purely functional package manager Nixpkgs. Nix packages collection NixOS. Nix based GNU/Linux distribution Hydra. Nix based continuous build and integration server Disnix. Nix based distributed service deployment Software available under free and open-source licenses (LGPL/X11)
30 References Nix package manager can be used on any Linux system, FreeBSD, OpenSolaris, Darwin and Cygwin Virtualization features can be used on any Linux system running the Nix package manager and KVM.
The Purely Functional Software Deployment Model Het puur functionele softwaredeploymentmodel (met een samenvatting in het Nederlands) Proefschrift ter verkrijging van de graad van doctor aan de Universiteit
Double-Take Replication in the VMware Environment: Building DR solutions using Double-Take and VMware Infrastructure and VMware Server Double-Take Software, Inc. 257 Turnpike Road; Suite 210 Southborough,
Best Practices Guide McAfee epolicy Orchestrator for use with epolicy Orchestrator versions 4.5.0 and 4.0.0 COPYRIGHT Copyright 2011 McAfee, Inc. All Rights Reserved. No part of this publication may be
This video will look the different versions of Active Directory Federation Services. This includes which features are available in each one and which operating system you need in order to use these features.
White Paper System Administration for the Intel Xeon Phi Coprocessor 1 Preface This document provides a general overview of system administration on the Intel Xeon Phi coprocessor. It is written with the
An Oracle White Paper June 2013 Oracle Real Application Clusters One Node Executive Overview... 1 Oracle RAC One Node 12c Overview... 2 Best In-Class Oracle Database Availability... 5 Better Oracle Database
WHITE PAPER: CA ARCserve Backup Network Data Management Protocol (NDMP) Network Attached Storage (NAS) Option: Integrated Protection for Heterogeneous NAS Environments CA ARCserve Backup: Protecting heterogeneous
Introduction By leveraging the inherent benefits of a virtualization based platform, a Microsoft Exchange Server 2007 deployment on VMware Infrastructure 3 offers a variety of availability and recovery
SECURE YOUR NETWORK WITH FIREWALL BUILDER firewall isolates a trusted, secure internal network from another network like the Internet, which is not regarded as either trusted or secure. Firewall Builder
A Survey on Virtual Machine Security Jenni Susan Reuben Helsinki University of Technology firstname.lastname@example.org Abstract Virtualization plays a major role in helping the organizations to reduce the operational
WHITE PAPER Addressing Virtualization and High-Availability Needs with Sun Solaris Cluster Sponsored by: Sun Microsystems Jean S. Bozman October 2009 EXECUTIVE SUMMARY Global Headquarters: 5 Speen Street
Contents Introduction...1 Overview of x86 Virtualization...2 CPU Virtualization...3 The Challenges of x86 Hardware Virtualization...3 Technique 1 - Full Virtualization using Binary Translation...4 Technique
Service Provisioning and Management in Virtual Private Active Networks Fábio Luciano Verdi and Edmundo R. M. Madeira Institute of Computing, University of Campinas (UNICAMP) 13083-970, Campinas-SP, Brazil
GE Measurement & Control Remote Comms System Installation and User Reference Guide Contents BENEFITS OF REMOTE COMMS SYSTEM... 1 HOW THE REMOTE COMMS SYSTEM WORKS... 3 COMPONENTS OF REMOTE COMMS SYSTEM...
NetVault, NDMP and Network Attached Storage Simplicity and power for NAS Written by Adrian Moir, Dell Scott Hetrick, Dell Abstract This technical brief explains how Network Data Management Protocol (NDMP)
Securing Data at Rest with Encryption Ken Beer and Ryan Holland November 2013 (Please consult http://aws.amazon.com/whitepapers for the latest version of this whitepaper) Page 1 of 15 Abstract Organizational
Special Publication 800-125 Guide to Security for Full Virtualization Technologies Recommendations of the National Institute of Standards and Technology Karen Scarfone Murugiah Souppaya Paul Hoffman NIST
The Incremental Advantage: MIGRATE TRADITIONAL APPLICATIONS FROM YOUR ON-PREMISES VMWARE ENVIRONMENT TO THE HYBRID CLOUD IN FIVE STEPS CONTENTS Introduction..................... 2 Five Steps to the Hybrid
WHITE PAPER Virtualizing UC: Reaping the Benefits and Understanding the Issues for Real-Time Communications Sponsored by: Avaya Abner Germanow October 2009 Jonathan Edwards PROBLEM DEFINITION Global Headquarters:
Vijeo Citect run as a Windows service December 2013 / White Paper Olivier Vallee Validation Specialist Make the most of your energy Summary Introduction... p. 01 Session Isolation... p. 04 OPC Servers...
Sophos Endpoint Security and Control on-premise installation best practice guide Endpoint Security and Control 10 Enterprise Console 5 Document date: May 2014 Contents 1 About this guide...3 2 What software
Execute This! Analyzing Unsafe and Malicious Dynamic Code Loading in Android Applications Sebastian Poeplau, Yanick Fratantonio, Antonio Bianchi, Christopher Kruegel, Giovanni Vigna UC Santa Barbara Santa
HP D2D NAS Integration with HP Data Protector 6.11 Abstract This guide provides step by step instructions on how to configure and optimize HP Data Protector 6.11 in order to back up to HP D2D Backup Systems
EOS: The Next Generation Extensible Operating System Performance, resiliency and programmability across the entire network are now fundamental business requirements for next generation cloud and enterprise