Technical report SPSS and Large Datasets: Two-Tier Client/ Architecture Table of contents Introduction................................................................ 1 Benefits of a distributed architecture in analytical applications...................... 1 The SPSS architecture........................................................ 2 Data scalability............................................................. 3 Data access................................................................ 4 Local mode versus distributed mode........................................... 7 Additional benefits and considerations......................................... 8 Conclusion................................................................. 9 Product availability......................................................... 10 SPSS is a registered trademark and the other SPSS products named are trademarks of SPSS Inc. All other names are trademarks of their respective owners. 2005 SPSS Inc. SDAAWP-0105
Introduction How can you increase productivity and facilitate decision making when analyzing data across an enterprise? One way to do so is to run SPSS for Windows, a comprehensive statistical software package, on a server. If your organization already uses SPSS, you ll appreciate the same ease of use and interactive interface you already know. If you re new to SPSS, you ll discover the powerful combination of two products: SPSS for Windows SPSS Inc. s flagship software for in-depth data exploration, analytical reporting, and modeling encompasses both the stand-alone software and the client SPSS this server-based, multi-platform software delivers enterprise-strength scalability, additional tools, security, and enhanced performance Client/server architecture enables you to make better decisions faster by analyzing massive, shared data sources that are centralized and can be stored on many types of servers. You can operate SPSS through its familiar desktop interface while leveraging high-speed Windows, Sun Solaris, Red Hat Linux, and IBM AIX server systems for computation and data-intensive operations. The client/server architecture partitions applications across desktop and server machines to balance usability and interactivity with speed and scalability. Additionally, each SPSS add-on module (such as SPSS Tables ) has a corresponding server version. You can also integrate a relational database management system (RDBMS) through SQL pushback, which enables you to leverage database investments. Benefits of a distributed architecture in analytical applications With the enormous influx of information gathered today by corporate, government, academic, and other institutions comes more opportunities to use this information to drive and expedite decisions at all levels in the organization. And, organizations increasingly expect to analyze their larger datasets quickly. Client/server architecture helps you achieve these goals. With it, you can: Perform analysis more quickly. With large datasets, preparation and analysis often take hours or even days to complete on desktop machines. By using enterprise-level, server-grade hardware that is shared among analytical users, your organization can dramatically reduce the time-to-decision, and free up desktop and network resources. Reduce network traffic. Desktop users of analytical software can severely decrease performance for entire networks of users as they transport large datasets across networks, even if the bandwidth requirements for these other users are modest. Analytical servers, on the other hand, are often located close to database server machines and are configured with higher bandwidth connections. You can achieve optimal performance by running network-intensive operations, such as database reads, on SPSS rather than on a desktop computer. By partitioning the application to minimize network traffic to desktop machines, SPSS helps eliminate or reduce bottlenecks in your analytical process. Centralize monitoring and control. Like transactional systems, centralized monitoring and control facilities can be used with server-based products to improve your IT department s ability to balance processing power among you and your peers. SPSS and Large Datasets: Two-Tier Client/ Architecture 1
The SPSS architecture In a standard two-tier architecture using the industry-standard TCP/IP network protocol, it is appropriate to consider the SPSS front-end application, or interface, as the client, and the SPSS back-end application as the server software. (Note: SPSS may run on a Novell network, but only if the network is also running the TCP/IP protocol.) The front-end application manages the user interface and displays results. The back-end manages access to and the computation of the data being analyzed. Generally, the front-end application does not directly use the record data. This reduces network traffic between the front-end and back-end to make use of the back-end server processor. You can run SPSS as a local application, which is referred to as local mode (Figure 1). In addition, if your site has installed an SPSS, you can also run in the distributed mode (Figure 2). You can quickly change between modes as necessary. Figure 1 SPSS local mode Syntax Output Script SPSS Data Figure 2 Tier 1 Tier 2 Syntax Output Analytical SPSS client/server architecture, distributed mode. Two-tier architecture is shown for simplicity. You can also set up the distributed-mode architecture as an N-tier structure. Script SPSS Client SPSS Data 2 SPSS and Large Datasets: Two-Tier Client/ Architecture
In order to connect the client to a remote server, users need to provide the following information: name name or IP address of the machine running SPSS Port number the port number to which the SPSS is connecting; defaults to 30XX (where XX reflects the version of server install; 12 for version 12.0, 13 for 13.0, etc.) User name account user ID; this account must exist on the server Password password associated with a user name Domain NT domain for which the user name account is valid; defaults to the domain used upon opening the client SPSS back-end server processes are launched and managed by a parent server, which runs as a Windows service or UNIX daemon and is normally started at the operating system's boot-up time. The parent server is responsible for accepting your login information and initiating SPSS. SPSS then runs with your rights and privileges. SPSS, however, cannot run alone; an SPSS client must drive it. Unlike transactional applications, each SPSS client is served by its own SPSS child server. This means, for example, when five users share an SPSS, there are five instances of the SPSS back-end running on the server computer. Each of these back-end processes runs independently and there are no assumptions made about sharing data or other resources. Since SPSS runs each session in a separate process, an individual user s preferences are secured when multiple users are using SPSS. If you re an experienced SPSS for Windows user, there s a minimal learning curve for operating SPSS in distributed mode. All of the familiar data, syntax, script, and output windows are present. The menus and toolbars are largely unchanged, as is the operation of the bulk of the user interface. The GUI, output editors, and presentation features continue to reside on the desktop. Plus, you continue to have a full copy of SPSS on your desktop, which you can run in local mode. Within SPSS, you have access to features and functionality that are only available in distributed mode. These include the ability to perform SQL data aggregation and sorting in the database prior to transferring data to SPSS. You can specify these operations through the SPSS interface. In addition, selected SPSS procedures can produce models that you can use to rapidly score new data via the SPSS ApplyModel transformation. Data scalability Much of the data read into analytical applications such as SPSS does not change in a typical session. SPSS data management efficiency in both local and distributed modes allows SPSS to handle minimal amounts of data when changing small amounts of data. Overall, this makes the data analysis process faster. Data access components (DACs) SPSS includes four DACs Text DAC, Excel DAC, SPSS DAC, and SQL DAC for reading data. The DACs have special client/server capabilities, most notably a user interface that detects the server file system and data access capabilities. All DAC server components can read incoming record data and turn them into records readable by SPSS. Additionally, all DACs produce SPSS syntax and can be used in syntax files and production jobs. More specifically, the Text DAC incorporates functionality from the client's Text Wizard, pairing it with a new server component and taking advantage of extensive performance improvements introduced in SPSS 13.0. The Excel DAC reads Microsoft Excel files and also has a user interface for selecting the Excel sheet, cell range, etc. Note that the Excel DAC can be used only on Windows servers. The SPSS DAC reads SPSS files (much like the GET FILE command in previous versions of SPSS). Finally, the SQL DAC is integrated with the Database Wizard on the client. This enables you to define data pulls that don t require manual creation of SQL statements. The SQL DAC supports the display of tables and fields from data sources installed on the server machine. SPSS and Large Datasets: Two-Tier Client/ Architecture 3
The virtual active file (VAF) The VAF consists of variable definitions with their associated data columns, and is stored as the initial data required, plus a series of Deltas that record changes to the data during the session. Deltas are internal SPSS data structures that store only the differences between themselves and the previous generation, without copying unaffected variables or columns. Actual Delta record storage resides in the temporary directory. The Data Editor window always displays the current view of the VAF. There are instances when copying your data and converting it into SPSS format, a process called caching the active file, is appropriate. If you are working with modest-sized data and if SPSS is repeatedly reading the data during the session, working with cached data is faster. However, when the datasets are large (millions of records and hundreds of variables), caching the data may be impractical because of space limitations in the temporary directory. For maximum flexibility when working with differently sized problem sets, you can cache the VAF at any time. This converts the entire VAF into a file in the SPSS temporary directory, combining the net affect of all Deltas. Once all of the Deltas have been applied to the temporary file, SPSS discards them. To further avoid cached copies of data, the ADD FILES and MATCH FILES commands, used to join two or more SPSS data files, have been optimized. The SORT CASES command, however, always produces a cache copy. Whenever possible, you should execute sorts within a database prior to extraction into SPSS. With SPSS, you can do this and aggregation tasks through the SPSS Database Wizard user interface. As such, the VAF architecture offers significant reduction of the disk space needed to run analytics. It enables you to easily create large SPSS jobs without creating entire copies of the input data, and while maintaining direct control to trade off disk space for performance. Data access SPSS uses standard SQL calls to access data in relational databases, as well as in some non-relational data sources. The Database Wizard supports remote data sources in its user interface. As you might expect, the user interface runs on the client, and operations on potentially large data tables run on SPSS. Setup processes for both the client and SPSS install the SPSS Data Access Pack, which is the universal SequeLink SQL driver, plus all DataDirect Connect drivers. These are combined into one pack to simplify installation. Detailed descriptions of both driver types are given below. Your site administrator can determine which technology is best suited for your operations. DataDirect Connect ODBC The first driver type, DataDirect Connect ODBC, is a comprehensive set of individual, database-specific wire protocol ODBC drivers that deliver reliable connectivity to all major databases across all major platforms. You can install the ODBC drivers on the client or server machine. Figure 3 (on the next page) shows SPSS configured with DataDirect Connect ODBC in distributed mode, and Figure 4 (also on the next page) shows the same in local mode. 4 SPSS and Large Datasets: Two-Tier Client/ Architecture
Figure 3 Tier 1 Tier 2 Tier 3 Analytical Data SPSS Client SPSS ODBC DataDirect Connect ODBC Transport DBMS SPSS client/server architecture with DataDirect Connect ODBC drivers. You can also set up the distributed-mode architecture as an N-tier structure, and data can be on the same server as SPSS. Figure 4 Data SPSS ODBC DataDirect Connect ODBC Transport DBMS SPSS local-mode architecture with DataDirect Connect ODBC drivers. SPSS and Large Datasets: Two-Tier Client/ Architecture 5
DataDirect SequeLink The second driver type is DataDirect SequeLink software, which is scalable, server-based middleware that supports a complete platform for data connectivity. SequeLink is tightly integrated with the SPSS database engine for maximum performance. With DataDirect SequeLink, your administrator can set up data access for one person, and everyone in your organization can use the same setup. It is also SQL based, and offers a universal SQL driver that works with all supported data sources. The SequeLink SQL driver has built-in transport software that communicates with the appropriate SequeLink, which runs on a remote machine. The SequeLink is database specific and is available for Oracle, IBM DB2, IBM Informix, Microsoft SQL, and other platforms. Figure 5 (below) shows an SPSS distributed-mode configuration using SequeLink. Figure 5 Tier 1 Tier 2 Tier 3 SPSS client/server architecture with an SPSS distributed-mode configuration using DataDirect SequeLink. Analytical Data SPSS Client SPSS ODBC DataDirect SequeLink DBMS SPSS Client SequeLink The SequeLink may run on any machine visible to the SPSS, but runs most efficiently on an RDBMS server machine. Communication between SequeLink SQL and SequeLink is optimized for large packet transfers across the TCP/IP protocol, which helps reduce access times to large tables. Database connectivity selection Site administrators and/or end users must decide which database connectivity, DataDirect Connect ODBC or SequeLink, to use for accessing their relational data sources. SPSS Inc. provides advice to assist in this selection through a document entitled SPSS Data Access Pack Installation Instructions. You can find this document on the SPSS installation CD. At the time of the publication of this paper, SPSS Inc. recommends DataDirect Connect ODBC drivers for most situations because they are easier to install and maintain, and usually offer better performance. DataDirect SequeLink, however, may prove more beneficial in some circumstances. It provides access to a wider range of data sources, more easily incorporates new data sources, and enables you to configure and manage data access from a single point of control through a centralized console. Note that these data access technologies are closed and can be used only with SPSS products. 6 SPSS and Large Datasets: Two-Tier Client/ Architecture
DataDirect Connect JDBC DataDirect Connect JDBC is a comprehensive set of Type 4 JDBC drivers that provide connectively to all major databases for the enterprise Java sm environment. Use DataDirect Connect JDBC only when your SPSS product relies on JDBC technology. Writing data back to databases To write data back to a SQL data source, use the SAVE TRANSLATE command. This function is available through the SPSS syntax. All DataDirect (Merant) drivers support the ability to write data back to the database using this command. Database Access Administrator The Database Access Administrator offers significant benefits in a client/server environment. Designed to offer multipletier advantages, it enables administrators to define metadata that is inherited by all users of the SQL DAC (described on page 3). Administrators who want to hide irrelevant database tables or rename tables or fields can run the Database Access Administrator and save enterprise-level metadata on SPSS, making it automatically available for all users. Local mode versus distributed mode Every SPSS user has the ability to run SPSS in local mode. If your site purchases SPSS, then you can also choose to run in distributed mode. Here are some guidelines for deciding the mode in which to operate. You must run SPSS in local mode when: Your datasets are stored on a local drive that cannot be shared across the network You are not connected to the network No SPSS s are available on the network, or the available SPSS machines do not have the SPSS add-on modules you require You should consider running in local mode when: Your datasets are small, for example, less than or equal to 2MB in size, and are accessible from your desktop Your computation runs at an acceptable speed on your desktop PC Local resources used by SPSS will be such that other tasks can be performed at acceptable performance levels during SPSS data preparation and analysis processing You must run SPSS in distributed mode when: Your desktop PC has insufficient memory, virtual memory, or temporary disk space to run your analyses Your Windows PC does not have the required database access software to communicate with the database Your network administrator does not permit you to download large data tables to your Windows PC You should consider running in distributed mode when: Your datasets are large and stored in a commercial database Your computation runs slowly on your Windows PC, and the machine running SPSS is significantly more powerful You require local desktop resources for other tasks to be performed during data preparation and/or analysis SPSS and Large Datasets: Two-Tier Client/ Architecture 7
Additional benefits and considerations Production mode The SPSS Production Facility is enhanced to run both local- and distributed-mode jobs in Windows environments. Job scheduling enables you to simultaneously run multiple production jobs. To support job scheduling, use command-line arguments to designate the server. You can also use the Production Facility s user interface to designate server machines. SPSS Batch Facility (SPSSB) SPSSB is a batch-processing utility included with SPSS and is an alternative way to use the power of SPSS. SPSSB runs SPSS jobs (syntax files) without requiring user intervention or an SPSS client. It produces output as XML, HTML, or text, rather than as SPSS Viewer output. SPSSB is helpful if users at your site often run the same time-consuming SPSS tasks, such as new variable derivation or weekly reports. SPSSB is a streamlined application specifically designed for batch production in a server environment. SPSSB can be used in Windows or UNIX environments, and is the only batch processing option available for UNIX sites. Sites with SPSS for Windows clients can also use the SPSS Production Facility (see above) for automated production. However, the Production Facility requires that the client is live and connected to SPSS at all times during the process. SPSSB, on the other hand, can run independent of the client, enabling literal lights-out batch processing ensuring that data and reports are ready for use the next day. In general, it is best to use SPSSB when you: Want to automate data preparation, such as new variable derivation within a large dataset Want to run production jobs independent of the client Want to automate production of reports Have established reporting requirements that can be met with standard production jobs Don t need to do production formatting of tables In general, it is best to use the SPSS Production Facility when you: Want to automate production of reports in a Windows desktop environment Want to modify output with looks and scripts Have varying and frequently changing requirements for reports Require output in the SPSS SPO file format Multiple instances of SPSS You can run multiple simultaneous sessions of SPSS on the same PC, in local mode, or in distributed mode, or any combination thereof. 8 SPSS and Large Datasets: Two-Tier Client/ Architecture
SPSS administration The SPSS administrator s role is to: Install SPSS Install user accounts on the SPSS machine as necessary and configure them for individual or group use, as well as to create accounts for end users* Provide information to SPSS users to assist them in connecting to SPSS, using data source names, understanding their rights to use data on the server, etc. Install and configure database connectivity software on SPSS Monitor use and resources Renew SPSS licenses as needed SPSS includes an administrator utility for monitoring and managing SPSS usage on the server. Through this utility, you can set most important SPSS customizations, including: SPSS temporary directories set a global (or user) environment variable called SPSSTMPDIR Home directories configurable as part of users system accounts, and applies equally to SPSS and other applications Log files SPSS maintains a log of user activity. You can configure its location. Port numbers configurable in the unlikely event that the SPSS needs to use a different port number ODBC data sources configure and test for your user base User priorities assign higher priority to select users, which prioritizes their analyses over those of other users, so that they do not have to wait until all preceding processes are complete Hard disk locations for temporary directories assign individual users or groups to unique disks for use in reading and writing temporary files, reducing bottlenecks that can occur when an entire user base shares one hard disk As discussed above, the SPSS administrator utility involves software installation on either the SPSS machine and/or the database machine. The administrator must know what data SPSS users will require access to, and then must configure and test ODBC-system DSNs for each data source. You may install ODBC-user DSNs for individual user accounts to give access to specialized data sources. SPSS and SPSS can be installed in any order; there are no installation dependencies between the two as long as they run at compatible version levels (e.g., SPSS 13.0 must run with SPSS 13.0 for Windows). Data security Security for SPSS primarily involves file and database security. SPSS is subject to file read-and-write privileges on both the client and server. Access to database tables is subject to user/password privileges in the database itself. Secure sockets layer (SSL) encryption of the traffic between client and server is possible and ensures that all communication between the two is unreadable by third parties if intercepted. Conclusion When you combine the strength of world-class analytical tools and techniques with the flexibility and speed of server functionality, you have a powerful solution for supporting better decision making throughout your enterprise. SPSS client/server architecture delivers on its promise of truly scalable statistical data preparation and analysis without sacrificing ease of use and familiarity. * It is possible to create accounts to be shared by groups of users, and it is often the case that users PCs will be in the same Windows domain as SPSS. In that case, no account setup is required because login details (user name and password) will be the same as users Windows network logins. SPSS and Large Datasets: Two-Tier Client/ Architecture 9
Product availability Hardware and operating system support SPSS client platforms (as of version 13.0): Windows XP Windows 2000 Windows Me Windows 98 SPSS platforms** (as of version 13.0): Windows 2003 or 2000 Sun Solaris 2.8 or later IBM AIX 4.3.3 or later Red Hat Linux 8.0 or later SPSS modules Each SPSS client add-on module has a corresponding SPSS module: SPSS Regression Models SPSS Advanced Models SPSS Classification Trees SPSS Tables SPSS Exact Tests (available for Windows only) SPSS Categories SPSS Trends SPSS Conjoint SPSS Missing Value Analysis SPSS Maps SPSS Complex Samples ** Platform availability subject to change upon release. To learn more, please visit www.spss.com. For SPSS office locations and telephone numbers, go to www.spss.com/worldwide. SPSS is a registered trademark and the other SPSS products named are trademarks of SPSS Inc. All other names are trademarks of their respective owners. 2005 SPSS Inc. All rights reserved. SDAAWP-0105