SmartArrays and Java Frequently Asked Questions What are SmartArrays? A SmartArray is an intelligent multidimensional array of data. Intelligent means that it has built-in knowledge of how to perform operations on the data. Multidimensional means that the data is organized along zero or more dimensions. A vector is a one-dimensional array, a matrix has two dimensions, a 3-cube has 3 dimensions, a 4-cube has 4 dimensions, etc. You can also have scalar arrays, which are a single element of data with zero dimensions. These arrays are created and managed by SmartArrays, a software component that creates arrays and performs manipulations and calculations them. This component is a native library that is accessed from Java through a Java class SmArray. It can easily be used from server side or client side Java code. What are the benefits of SmartArrays with Java? SmartArrays are a natural complement to Java, taking advantage of the things that Java does well, and expanding Java s power in new ways. Multidimensional arrays: SmartArrays makes it easy to represent data cubes such as are used in OLAP and multidimensional analysis applications. Rapid Development: SmartArrays provides high-level software building blocks that work on entire arrays of data. These can be rapidly assembled to express algorithms that otherwise would require much more thought and more code in straight Java. Fewer bugs, easier maintenance: The logic inside SmartArrays is robust and reliable, and applications built with SmartArrays are easier to modify and enhance. High Performance: SmartArrays logic is written in optimized C++, and uses sophisticated algorithms. Calculations can run many times faster than the same operations expressed in straightforward Java programs.. What kinds of applications are array-oriented? Actually, arrays are everywhere once you learn how to see them. A spreadsheet is a matrix. So is a table in a relational database. A text file is a vector of bytes.. A photograph represented as a bitmap is can be thought of as a matrix of pixels, or a 3-cube of rows, columns, and red-greenblue levels. A movie is a cube of photographs. SmartArrays and Java - FAQ James G. Wheeler 2/5/02 page 1
Typically, the more experience you have with using arrays, the more readily you will see places to use them. Getting the hang of array-oriented thinking doesn t happen overnight, any more than object-oriented thinking comes immediately to a programmer used to procedural programming. But the more you use SmartArrays, the easier it will be for you to spot the right place to use arrays in your programs. As you do, you ll find that your computations that used to seem complicated now appear simple, and you will attempt more sophisticated calculations that seemed too hard before. In practice, one of the most natural places to use SmartArrays is with financial or marketing data, where one often needs to organize a large volume of data elements along multiple dimensions. For example, sales information might be represented as a 4-cube organized by product, customer, date, and salesman. Putting this information into a 4-dimensional array makes it easy to create reports viewing the data along any two axes (a practice sometimes called spinning the cube ). SmartArrays enables applications to go much further, however, performing calculations of any kind on the data in the cube, projecting future sales, etc. What is the difference between SmartArrays and a Java array class library? Other Java class libraries exist that provide some form of multidimensional arrays. The standard Java Collections framework provides for arrays of Java objects, and IBM s AlphaWorks has a Numerically Intensive Java package that includes arrays and implements a suite of standard linear algebra subroutines. Such array classes can be useful, but are no substitute for SmartArrays. As a high-performance array engine, SmartArrays provides unique capabilities that are not available with other products. Among the important distinctions are: 1. The SmartArrays engine is not written in Java, but in highly optimized C++. As a result it does not suffer from the performance problems of bytecode-interpreted Java. This can make a profound difference when calculations operate on large quantities of data. 2. SmartArrays can be used with other languages besides Java, including C, C++, Visual Basic, Delphi, etc. Multiple object models are supported, including COM/DCOM and CORBA, in addition to JNI (Java Native Interface) calls. 3. SmartArrays can be configured to run as a server that can provide array services to multiple clients at once, and it is inherently networkable, allowing a Java program on one machine to manipulate arrays of data on another. 4. SmartArrays can handle very large arrays. The current version handles individual arrays up to 4 gigabytes in size, with as many as 255 dimensions, and this limit will be raised as 64-bit machine architectures come into production use. Memory used by SmartArrays is separate from the Java Virtual Machine s memory manager and is managed in a way that is tuned for large arrays. 5. SmartArrays provides a complete set of array-oriented primitive functions, including arithmetic and math functions, structural manipulations, sorting, searching, etc. How are SmartArrays made available to my Java program? Each SmartArray is created, owned, and managed by a SmartArrays engine. Under Windows, the engine is a DLL (dynamic link library) that can be called from any program. For a Java program, the engine is called by the JVM (Java Virtual Machine) that executes Java programs on the machine where SmartArrays is installed. SmartArrays and Java - FAQ James G. Wheeler 2/5/02 page 2
This detail is invisible to your Java code, because the functionality of the SmartArrays engine is wrapped with Java classes. Your program imports these classes and calls them as it would any Java class. The actual calling mechanism is through JNI (the Java Native Interface) which is the standard protocol for calling external libraries from Java. What does a SmartArray look like to Java? The fundamental Java class for working with SmartArrays is SmArray: This represents a single SmartArray. It contains a link to the actual data (stored in memory owned by the SmartArrays engine), and implements more than 150 built-in array methods that can operate on that data. What kinds of operations can SmartArrays do? SmartArrays provides a comprehensive suite of built-in primitive functions, each of which works on entire arrays of data. These are called primitives because they can be combined together in an infinite variety of ways to perform more complex calculations and data manipulations. Besides the primitive functions, SmartArrays has a database interface for creating arrays with data from a database, a file interface for moving data into and out of files, and a formula engine that executes Excel-style formulas on individual data cells in an array. SmartArrays also provides a facility for plug-in extensions, so new custom operations on arrays can be added. How to I create arrays of data? Here s a simple example that creates a one-dimensional array (vector) containing the number s100, 200, and 300: SmArray my_array = SmArray.vector( 100, 200, 300 ); The following creates a new scalar, containing the integer value 99: SmArray ninetynine = SmArray.scalar(99); This creates a new floating-point SmartArray vector from data in a Java vector: float[] data = { 1.0, 1.5, 2.0, 2.5, 3.0 }; SmArray smartdata = my_server.newfloatvector( data ); The following creates a 3-cube with a specified shape. The shape of {3,4,5} means that the array will have 3 planes by 4 columns by 5 rows. The data supplied is repeated to fill the entire array, resulting in each row containing the same 5 values: int[] shape = {3,4,5}; int[] data = {100,200,300,400,500}; SmArray my_cube = new SmArray( shape, data ); How do I calculate with arrays? Here are some examples, using the array data created above. // multiply all the numbers in <my_cube> by 1000 SmArray data2 = mycube.times( 1000 ); SmartArrays and Java - FAQ James G. Wheeler 2/5/02 page 3
// calculate the sum of <my_cube> along the rows (last axis); yields // a 3x4 matrix. SmArray sum = my_cube.reduce( Sm.plus ); // calculate the maximum value of each row of data; 3x4 matrix result SmArray max = my_cube.reduce( Sm.max ); Notice how we can use different functions with the Reduce operator: The auxiliary class Sm contains mnemonically named constants for all of the built-in methods Sm.plus to find the sum, Sm.max to find the largest. This same pattern works with any of the calculation methods. // catenate matrix <max> onto my_cube, making it a 3x4x6 cube my_cube = my_cube.catenate( max ); Catenating a 3x4matrix onto a 3x4x5 cube, producing a 3x4x6 cube How to I move data between SmartArrays and Java? Class SmArray provides a set of handy methods for initializing arrays with Java data, and moving the data into and out of arrays. A few examples: int[] jdata = new int[]{ 4, 5, 2 1 }; // regular Java array SmArray sdata = new SmArray( jdata ); // into a SmArray int[] jdata_reversed = sdata.reverse().getints(); // back into Java int v0 = sdata.getint( 2 ); What kinds of data can I store in a SmartArray? SmartArrays can hold numeric data in several forms: boolean (one bit per data element) integer (32-bits) real (64-bit) Character data can be stored as: 1-byte characters 2-byte Unicode characters atomic Unicode strings. These are sequences of character that are treated as a single element of data. You can also create arrays of arrays, where each element is another array. These are called nested arrays. SmartArrays is different from Java or C++ in that it automatically converts data to an appropriate type. Thus, you can add a boolean matrix to an integer matrix and get an integer result. You can multiply two integer arrays together and the result will normally be integer. If, however, any of the multiplications overflows (the result is too large to store in a 32-bit integer), SmartArrays will automatically produce a floating-point array that contains the correct answer. SmartArrays and Java - FAQ James G. Wheeler 2/5/02 page 4
How can I use SmartArrays in server-side programs? When the SmartArrays server DLL and Java classes are installed on a server machine, any server-side Java program can use SmartArrays standalone applications, servlets, custom Enterprise JavaBeans, etc. Client PC Server Web brower client page Network or Internet Java Servlet SmartArrays Database Server Array Array Using SmartArrays on a server, controlled by a client-side applet How can I use SmartArrays in standalone client programs? The SmartArrays server DLL can also be installed on a client PC and create arrays locally. The DLL is small enough (about 400Kbytes) that it can be downloaded on demand from a server. How can I use SmartArrays with database data? SmartArrays has a built-in interface to SQL data sources. This provides all of the features needed to work with a database, including metadata queries ( what are the columns of this table? ), which create arrays containing the result information. It is possible to perform queries on the database, with the entire result materializing as a SmartArray. These arrays can then be manipulated in ways that are clumsy or slow using SQL. You can also post data from a SmartArray into a database. What platforms does SmartArrays run on? SmartArrays currently runs 32-bit Windows (2000, NT, 98, 95); a Unix version is planned for the leading Unix environments (AIX, Solaris, HP/UX, Linux). Because SmartArrays is written in pure C++ and does not use third-party class libraries, it is very portable. Copyright 2000, SmartArrays, Inc. Java is a trademark of Sun Microsystems, Inc. SmartArrays is a trademark of SmartArrays, Inc. SmartArrays and Java - FAQ James G. Wheeler 2/5/02 page 5