Reference Architecture

Transcription

1 Reference Architecture Or get in touch: 1

2 Introduction This reference describes an ideal Linux infrastructure designed to cover the majority of application deployments. It is infuenced heavily by our experience in working with both large and small customers, and represents what we believe to be current best practice. We primarily use open-source and free tools, believing them to be the best choice in most cases, however some are also available under commercial support from their vendor if that is required. The approaches described here are applicable whether hosting on co-located servers, dedicated servers from a managed service provider, or using cloud servers. We believe that good infrastructure is: Scalable able to grow horizontally or vertically as required by the operational demands put on it by the business. Standards-based uses standard protocols, software and processes so as to be unsurprising to anyone working on it. Or get in touch: info@scalefactory.com 2

3 Supplier agnostic not tied to products or services of any organisation (including The Scale Factory), allowing portability between hosting suppliers or service organisations. Cost efective no more expensive than it needs to be, either in capital or operational expenditure. Current using modern tools and techniques, though without being too experimental about it. Repeatable easily repeated, so new server builds or secondary datacentre rollouts are straightforward. Available meets availability requirements for the business. Performant delivers resources quickly and efficiently. Responsive to change can deal with changing business requirements rapidly. Manageable easy to operate centrally, and without the requirement of heavy staffing levels. Operationally visible provides monitoring, logging and Or get in touch: 3

4 other tools so that it's clear what the platform is doing. Secure protected from external and internal attackers. Auditable transparent in process, and easy to see who has changed what and when Every design decision we make is pursuant to one or more of these goals. Or get in touch: 4

5 Basic Design The standard Scale Factory architecture is based around one or more hosting sites. Each site contains a set of supporting infrastructure for central control and observation purposes. This supporting infrastructure enables the operation of service hosts (either physical or virtual) arranged into clusters. These clusters run applications which are then arranged into platforms. Multi-site operation In this fgure, there are three sites dc1, dc2 and of1. Each of these contains support services. The site dc1 houses live and Or get in touch: info@scalefactory.com 5

6 staging platforms, which contain web and database services. There are no cross-site dependencies each site can work in isolation from the others. The live platform in dc2 might be a replica of that in dc1, with some global load balancing solution balancing traffic between the two for reasons of availability or geographic distribution. Or get in touch: info@scalefactory.com 6

7 Location The Scale Factory Reference Architecture is designed to be agnostic of where it is hosted, and can be built out in colocation suites, on managed servers or in fully cloud-hosted environments. Depending on your application's constraints, one or more of these options may not suit. If building out an infrastructure on your own hardware in a colocation suite, you get the highest level of control over how things are put together, and this may be desirable in the case of specialist application requirements, or for reasons of compliance. With this level of control comes an increased level of responsibility. If a RAID array throws a disc in the middle of the night, you'll need to be monitoring for and responding to that eventuality in some way. When co-locating, there will be a high up-front capital expenditure as you must buy servers, storage, network devices and cabling. Power constraints must be taken into account a modern rack is likely to run out of power before it is full and lower power devices are often more expensive. Server costs are often surprising to people who are used to buying consumer-grade hardware per-server costs of up to 4000 are Or get in touch: info@scalefactory.com 7

8 not uncommon. Lead times on hardware orders can sometimes take you by surprise too it's not completely unknown for a server order to take six weeks to be fulflled, particularly when adverse weather conditions cause RAM or hard disc manufacturing difficulties. An alternative to colocation is to take servers from a managed service provider. These organisations have hardware and network specialists on staf, and will take responsibility for keeping servers up, replacing failed discs, etc. In exchange for this, you may lose an element of control things like network layout and server specifcation are likely to be constrained to some extent. Managed server hosting costs vary widely across the industry, and most are open to negotiation. Some hosts charge higher setup fees in exchange for a lower monthly rate. Some ofer discounts if you pay for a whole year up-front. Providers who can sell access to both physical servers and virtual (or cloud) servers, with a private network between the two are ideal. Some have short billing terms on their cloud servers, allowing you to add and remove capacity as required by seasonal demand. Or get in touch: info@scalefactory.com 8

9 Fully cloud-hosted solutions (such as Amazon EC2 or Linode) are currently very popular. They are best suited to applications whose components all scale horizontally, and/or have extremely peaky levels of demand. Applications which require predictable performance can be problematic on cloud servers, as sharing resources with other customers means you get no guarantee of resource availability. The cheaper, smaller instances can be heavily CPU contended at times, and larger instances are more expensive month-onmonth than equivalent dedicated servers with a managed hosting supplier. We usually recommend that organisations who run applications centred around standard relational databases do so from managed or co-located hardware, but consider using cloud servers to deal with peaks in demand or to accelerate highly parallelisable jobs like data processing. Or get in touch: info@scalefactory.com 9

10 Supporting Infrastructure In a Scale Factory architecture, we install supporting infrastructure at each site. These provide services to the rest of the network, including confguration management, monitoring and backups, as well as central identity, time and naming services. This simplifes server management and ensures consistency across the network. The following diagram shows how individual server nodes use the supporting services. Or get in touch: info@scalefactory.com 10

11 Core servers are provided in a high-availability pair, with some services provided in an active/active model, and some in active/passive using a foating IP address: DHCP and TFTP services are only required in co-located sites where network confguration and automated system install is not performed by the supplier. Package repository mirrors may not need to be provided locally except as an optimisation in larger networks, or where more strict change control is a requirement. Or get in touch: info@scalefactory.com 11

12 Or get in touch: 12

13 Naming Sensible naming of hosts and services is more important than it appears. A good naming scheme represents the location and purpose of a device, as well as alluding to its membership of a platform, cluster or service tier. Poor or badly-implemented naming leads to administrative complexity, and increases the likelihood of operator error. We defne a consistent naming standard for all infrastructure components, ensure all hosts have forward and reverse DNS mappings and confgure them to report their hostname correctly. As DNS is laid out in a hierarchy, we assign a DNS search path so that inside any platform, referencing a hostname like 'db' will always fnd the most local service by that name for example in the 'live' platform, 'db' always means the 'live' database. This minimises the number of changes that need to be made to confguration fles between platform. Or get in touch: info@scalefactory.com 13

14 Identity Identity management is crucial to the operation of even a small systems infrastructure. Identity management systems are used to authenticate and authorise users. Authentication is used to establish that a user is who they claim to be. The most obvious example of this is the use of passwords. Authorisation is the system of checks that ensure your authenticated user can only access the resources and privileges that they need to do their jobs. We use 389 Directory Server 1 to provide central identity services to our infrastructure. All authentication and authorisation tasks can be performed this one system, which means each user only needs to learn one set of access credentials for all systems in the organisation. This includes access to servers, VPN services, web administration panels and so forth. 1 Or get in touch: info@scalefactory.com 14

15 Centralised identity management also means that when new employees start, or old employees leave, there is just a single place where their credentials need to be registered or removed, keeping complexity to a minimum. Administrative users have standard user accounts like everyone else. Their superuser access is then limited to the minimum set of privileged activities required to do their job. This has the advantage that there is no need to share account passwords. The root password can then be locked away in case of emergencies. Activities performed by these privileged users can then by logged against their identity for audit purposes. Batch processes and automated jobs are also assigned their own users, and have minimal access to superuser resources just as a human user would. System users (UIDs below 1000) are still managed in password and group fles. Or get in touch: info@scalefactory.com 15

16 Configuration Management When you install an operating system on a host, only the very basics of that environment are confgured. Before that host is usable in any real sense, other software needs installing, confguration fles need to be updated, and services need to be started. The traditional, naïve approach to this problem is to make these confguration changes by hand. This approach doesn't scale well to an infrastructure of more than a couple of servers, and isn't advisable even then. We use Puppet 2 to automate this type of confguration. Given a description of the desired server state, Puppet installs packages, updates confguration fles and starts services in order to reach that state. Throughout the lifetime of a server, other administrative changes are occasionally required. Puppet can make these too, and will only make the minimum set of changes required to reach the new desired state. Puppet's confguration is stored as plain text fles, and we use a version control system to manage these. As well as the 2 Or get in touch: info@scalefactory.com 16

17 benefts this lends to collaborative working, this also means all infrastructure changes beneft from an automated audit trail, showing which individual made changes and when. Confguration management minimises diferences between diferent systems and environments by using a standard set of tools for building all environments. It is an important part of ensuring parity between development and production environments. Disaster recovery and host rebuild becomes extremely straightforward when no manual confguration activity is required. All Scale Factory infrastructures are deployed from a common base set of Puppet manifests. These undergo constant development, and beneft from an automated test suite. We make then available to customers under a license which permits use and modifcation, but not redistribution. Or get in touch: info@scalefactory.com 17

18 Availability For most infrastructures, availability is important. In the most straightforward view of the world, this simply means that in order to cope with failure you need at least two of everything. For services which maintain no local state, such as web or content servers, it is generally possible to place two or more of these behind a network load balancer. In the case of services which hold their own state data, most provide their own approaches to high availability. For example, MySQL or SOLR database hosts can be confgured to replicate to one or more slave hosts. Memcache servers can be run as a cluster, as long as your application has a list of every node in the cluster. Where a service or resource can only exist on a single host at a time, cluster software exists to allow multiple hosts to negotiate with each other for the right to own that service or resource, and we regularly do this to allow fail-over of master IP addresses for services. Finally, application code should be able to cope gracefully if some resources are missing. Services should not need to be restarted if resources migrate to other hosts in a failover group. Or get in touch: info@scalefactory.com 18

19 High availability can come with associated cost, and so we look at each system's availability requirements in turn. For example in a standard Scale Factory infrastructure, we ensure that confguration management, identity, time and naming services are highly available as their unavailability can afect live service. Observational services such as monitoring and logging can run without failover to keep costs down, and largely without afecting live system availability. We use tools from the Linux Virtual Server Project 3 to provide IP load balancing and failover. In addition, in the case of load balancing, we can use load balancing services provided by our hosting providers, or managed hardware load balancers in the case of applications requiring a throughput greater than 10Mbps. 3 Or get in touch: info@scalefactory.com 19

20 Backups and Disaster Recovery Every infrastructure has important data which needs to be reliably backed up. For most web applications, this tends to be application code, database content and media assets. We design infrastructure so that the data you want to back up is held in a small number of well-defned places. In our standard design, we provide at least one host with plenty of (RAID mirrored) storage at each site. We run Bacula 4 on this host, confgured to back up important data on a nightly schedule. This backup can then be replicated of-site, with encryption where required for security. We run backups in a way which minimises service disruption. For example, with MySQL databases, this usually means taking snapshot backups from an otherwise unused replication slave. A backup is only as good as your ability to restore it, and so we provide methods by which restoration tests can be performed. We design infrastructure in such a way that, in a disaster recovery scenario where bare-metal recovery is required (such 4 Or get in touch: info@scalefactory.com 20

21 as loss of a complete data centre), it is possible to get up and running again on new hardware using nothing but operating system media, and backups. Or get in touch: info@scalefactory.com 21

22 Monitoring A hosting infrastructure is a large and often complicated beast, and in an environment that can, thanks to fail-over, cope with individual node failure it isn't possible to tell how healthy the platform is by simply testing whether a website is available to the outside world. We solve this problem by installing a monitoring system. A good monitoring system can monitor the health and performance of individual applications; the host operating system; the network; and hardware devices. The data collected is then usable for capacity planning, availability monitoring, performance analysis and troubleshooting. We install Zabbix 5 for this purpose, and provide a set of custom templates and data collectors for all of our standard pieces of software. 5 Or get in touch: info@scalefactory.com 22

23 Log Aggregation Systems often generate a lot of log data, and attempting to troubleshoot problems in a distributed system without centralised log aggregation is a painful exercise. Individual application servers shouldn't need huge amounts of space for log storage, and should ideally keep log data for only a limited amount of time. We use rsyslog 6 to ship live logging data from individual hosts to a single log server, where administrators can log in and view the logging output of the whole platform in a single place. Centralised logs make for easier troubleshooting, and software such as Splunk 7 (not free) or Graylog 8 can make this troubleshooting easier by ofering an easy to use interface for querying log data. Automated fltering and querying can also drive data collection or monitoring events, to help improve operational visibility. We're currently working on adding Graylog to our standard platform design Or get in touch: info@scalefactory.com 23

24 Architecture As an infrastructure grows, it pays to have thought about the architecture of the systems and software in advance. Certain architectural decisions can limit the potential for growth by making scalability difficult. Diferent parts of a system scale at diferent rates for example, on a photo sharing site such as Flickr, the infrastructure responsible for storage of photographs will be the fastest-growing part of the system; storage for the comments will scale at a diferent rate (probably more slowly); and the system which stores a username/password pair for each user will scale more slowly still. Our standard infrastructure design allows us to build service clusters of diferent sizes with this sort of variable growth in mind, though typically an application needs development work in order to break it into components which can be scaled that way. We can ofer consultancy on this activity. Or get in touch: info@scalefactory.com 24

25 Security Diferent infrastructures require diferent levels of security, but we always work with a sensible minimum set of requirements in mind. Using either physical frewalls or per-host iptables rules (depending on what the hosting provider can ofer), we ensure that administration is only possible from local core servers, or any trusted networks connected to the infrastructure by VPN. This includes access to SSH as well as any internal web-based admin or monitoring dashboards. When services provide methods by which to control access, we ensure that this is granted to consumers of this service on a least privilege basis, granting only as much access is required, and no more. Human operators are treated in the same way, with superuser access being limited to the minimum set of permissions required by that user to do their job. More complex infrastructures may require more interesting frewall provision, and may require physical frewall devices to completely separate hosts from each other on the network. This will help inform the hosting location debate, and will need to be discussed during the architecture phase. Or get in touch: info@scalefactory.com 25

26 Use of confguration management tools and least-privilege superuser access, along with centralised logging and monitoring provides an audit trail for all system activities. Use of automated testing in combination with confguration management tools means system software can rapidly and easily be upgraded. Accurate development and staging environments mean these upgrades can be tested in a preproduction environment before they're pushed live, removing most of the risk surrounding upgrade activities. Or get in touch: info@scalefactory.com 26

27 Development / launch process As a code base grows, rigorous manual testing becomes impossible. When testing is seen as a large time overhead, it is often skipped in order to meet deadlines, and a code base without tests becomes a place where developers fear to tread, slowing future progress. Automated testing should be considered mandatory for any modern software project. Continuous integration where tests are run every time code changes can be tricky to set up, but pays dividends over time. In an ideal world, if you have a broad suite of tests in which you trust, code launches can be automated, and happen continuously whenever changes are made. In a modern web application, change should be considered the norm, not the exception and we attempt to engineer our systems to cope with this view. We make use of the Jenkins 9 CI tool to automate test and deployment, and provide auditable visibility of the process. The testing approach itself will depend on language environment, but the following broad categories of test will no doubt be relevant: Unit testing 9 Or get in touch: info@scalefactory.com 27

28 BDD style scenario testing Integration testing Performance testing Penetration testing As well as running tests, Jenkins can be used to trigger automated deployment of application software using tools such as Capistrano 10 or Fabric 11 which removes the margin for error that software deployment can otherwise entail. We follow many of the best practise guidelines described in Continuous Delivery 12 by Jez Humble and David Farley Or get in touch: 28