Understanding Rasters By Joseph Collins-Unruh

Transcription

1 Understanding Rasters By Joseph Collins-Unruh - 1 -

2 Table of Contents Introduction and Statement of Purpose...3 General Properties of Rasters...4 Height and Width...4 Data Type...4 Vertical Order...4 Endianness (Byte Order)...5 Raster Graphics...6 Bands...6 Color Model...6 Interleave...7 Geographical Rasters...9 Cell spacing...9 Nodata Values (Null Values)...9 Cell Origin...9 Raster Origin...10 Data Structures...10 Continuous...10 Classified...10 Compression...11 Run Length Encoding (RLE)...11 LZW Compression...11 Case Study: ENVI.hdr format...12 Header File...12 Binary File...13 References

3 Introduction and Statement of Purpose When I started working at Safe Software I had no idea what a raster Dataset was. I spent hours scanning the internet searching for a precise definition of raster. Unfortunately, resources were scarce and I never got beyond the basic definition, A grid of cells or pixels, with each pixel containing a single value. While simple enough, this definition taught me nothing about how those pixels were represented or what you could do with them. This guide is my attempt to save you trouble by giving you a solid reference to the terms and ideas involved with rasters. It is an introduction to what a raster is so that you can start using them without worrying about terminology. Although the document was designed for programmers I ve put a great deal of effort into making it understandable by everyone. Non-programmers who read this can safely ignore the points that refer to programming languages and techniques. Also, the ENVI.hdr format case study in the last section will be of little use to non-programmers. This is by no means a complete reference. It is comprised only of the main subset of what I was exposed to while working at safe software, augmented significantly by research conducted at the time of writing to ensure accuracy. There are certainly other raster implementations and models beyond what I describe here. However, the vast majority of raster formats that I worked with while at Safe Software could be described by this model or an extension of this model with additional aspects such as compression methods or color models

4 General Properties of Rasters Height and Width The height and width of a raster is, quite simply, the number of rows and columns contained in the grid. Height and width are sometimes referred to as lines and samples respectively. The individual entries of a raster are called cells or, in the case of raster graphics, pixels. Figure 1 Raster Data Corresponding Type C++ Data Type 8-bit unsigned unsigned char Integer 8-bit signed char (character) Integer 16-bit unsigned unsigned short integer 16-bit signed short integer 32-bit unsigned unsigned int integer 32-bit integer int 32-bit real/float float 64-bit real/float double Table 1 Common numeric raster data types Data Type A raster s data type is the type of information stored in each cell of the raster. Raster data types are almost always numeric but practically any data type is possible. The standard for representing numeric data types has the form: [allocation]-bit [type] Where the allocation is the number of bits that the data type occupies and the type is one of: integer, unsigned integer (integers with no negative values), or real (numbers with decimals). The table to the left shows many common numeric raster data types and the associated C/C++ data types. Vertical Order The vertical order of a raster specifies what row of the raster you start reading from. For a vertical order that is top-down you start reading across the top row of the raster. When you ve read all of the columns of the top row you move on the second row down, and so on. The bottom-up vertical order is exactly the opposite. Here, you read all of the columns from the bottom row first then move on to the second row up. Figure 3 Bottom-Up Vertical Ordering - 4 -

5 Endianness (Byte Order) The endianness of a raster describes how the raster data is stored in memory. Endianness is not a property inherent to rasters, but rather a consequence of working with rasters on different computer systems with different standards for storing numeric data. The two most common endianness are little endian and big endian. Little endiandiannes is sometimes called least significant byte and occurs most commonly on machines based on the Intel architecture. Big Endianness is sometimes called most significant byte and occurs most commonly on machines based on the Motorola and SPARC architectures. The technical details that are important for dealing with endianness programmatically will not be dealt with here. Programmers should already know the specifics of endianness and Non-programmers shouldn t worry about endianness as it isn t critical in understanding rasters

6 Raster Graphics The most common type of raster is a digital image or graphical raster. Raster graphics are a computers primary method for displaying images and pictures. Most of the terms covered here are not limited solely to raster graphics. The ideas behind terms like band and interleave are most easily represented in the context of images. Bands The number of bands that a raster has specifies the number of values for each cell or pixel in the raster grid. For example, a RGB color image includes one band for each color red(r), green(g) and blue(b). Another word that is commonly used for bands is layers. Color Model The color model describes how the color of individual pixels is determined from the values at that point in the different bands. For single banded color models like grayscale, the pixel value simply represents a range of light values. However, in order to produce colors, more complex models are required. Mono Color: The most basic color model for rasters is mono color. With mono color each pixel can be represented by a single bit. When the bit is 0 then the pixel is black; when set to 1 the pixel is white. Figure 4 Grayscale: The next color model is grayscale. Grayscale images can use any data type with the minimum value representing black and the highest value representing white. However, almost all grayscale image formats use 8-bit unsigned integers with some using 16-bit unsigned integers for a more precise representation of color. In figure 5, to the right, the 8-bit unsigned integer data type is used for simplicity. So, in this case black is represented by 0 and white by 255. (starts counting from 0) RGB (Red Green Blue): Figure 5 The RGB color model requires that the raster have 3 bands, with one band for each of the three RGB colors; Red, Green and Blue. Like Grayscale each color typically uses 8-bit unsigned integers to represent the color rage. The difference is that each cell ranges from 0 being black and 255 being the associated color. The resulting pixel uses additive color mixing to obtain the final color, as shown in Figure 6 figures 6 and 7. Additive Color Mixing Figure 7-6 -

7 RGBA (Red Green Blue Alpha): The RGBA color model is exactly like that of RGB, except there is an additional band representing the pixel s alpha value or transparency. For example, if the alpha value for a pixel is 0 then the pixel will not be visible at all and you should see what is behind the image at that pixel. If, on the other hand, the alpha value is 255, then the pixel will be opaque and it will appear as a RGB pixel would. CMYK (Cyan Magenta Yellow Black): Like RGB and RGBA, CMYK is another type of color mixing color model, except CMYK uses subtractive color mixing instead. RGB color is analogous to the mixing of colors of light, with black representing the absence of light. CMYK on the other hand, is analogous to the mixing of pigments, where black represents all pigments combined. Figure 8 Subtractive Color Mixing Interleave The raster s interleave specifies how the pixels and bands in a raster are arranged. Different interleaves allow you to arrange the data non-contiguously to optimize certain methods of access. Interleave is only relevant in multi-banded rasters; the cell values for rasters with one band are written sequentially. The main interleaves are BIL, BIP, BSQ and Tiled, which are defined below. To the right, Figure 7 is a 4x4 image which is the basis for demonstrating how each of the interleaves is organized with the Figure 9 Base image for interleave examples RGB color model. In each case the overall size of the image will triple since all three bands of each pixel will be shown. Also, the values should all be read from left to right, top to bottom. Figure 10: BIP Each block outlined in white represents one pixel. Each line is one line of the image. Band-Interleaved-by-Pixel (BIP): This is the most intuitive interleave, where for a given pixel, the values for all bands are side by side. This interleave is most efficient for determining all of the values for a given pixel since they are all side by side

8 Band-Interleaved-by-Line (BIL): With BIL each raster row is made up of 3 band rows, one for each color component. BIL interleave is most efficient for accessing all bands of a given row sequentially. Figure 11: BIL Each block outlined in white represents one band of one row. Each line represents one line of the image. Band-Sequential (BSQ) Interleave: With BSQ interleave the entirety of the first band is written first, followed by the second band, and so on. BSQ Interleave is most efficient for accessing each band sequentially. This was the most commonly used interleave that I encountered while working with geographic data since it efficiently accesses all of the values for a single band. Tiled Interleave: With this interleave a raster is divided into some number of equal sized blocks, each written sequentially from left to right and top to bottom. Typically, each tile is interleaved with BSQ. For the example on the right, figure 9 was divided into 4 2x2 blocks. Figure 12: BSQ Each block outlined in white represents one band of the image Figure 13: Tiled Interleave Each block outlined in white represents one 2x2 tile - 8 -

9 Geographical Rasters Next to image rasters, geographical rasters are the most commonly encountered type of rasters. A geographical raster essentially divides a real world place into a grid and assigns some element of the real world to each cell of the grid. Often the data that is stored is the elevation at the cells position, but it can be almost anything that varies continuously or discretely over a geographical range. Other common geographical rasters measure temperature, soil composition or even land ownership. Cell spacing A raster s cell spacing describes the area that each cell covers in the real world. This cell spacing indirectly determines the smallest size an object can be and still be visible in the raster. For example, if a raster s cell spacing measures 3 meters in the x and y direction for a total of 9 square meters, in order for the feature to be observable it must take up most of the area of the cell or be at least 4.5 meters square. A raster s cell spacing is often described as the resolution of the raster since it determines how much detail can be observed. It s important to note that there is no requirement that the area represented by cell has to be square. There are many formats, such as GeoTiff and ENVI.hdr, which is described below, where the area covered by a cell can be defined as rectangular. Nodata Values (Null Values) Often when measuring values over a geographical range it is not possible to get data for every cell of the grid. For example, when measuring soil composition in the vicinity of a lake it s often not possible or relevant to get the data from below the surface of the lake. In those cases, you would enter a nodata value in that position. Typically, if a raster format supports nodata values it will reserve one extreme value that is rarely used to be the nodata value. For example, USGS DEM format uses , which is the smallest possible integer value as its nodata value. Nodata values are often used in images as well, for an example of this all you have to do is look at your computers desktop. There, the icons only have image data in some areas. All of the areas where you can see the background visible within icon, like the middle of the e in the Internet Explorer icon, are actually nodata values. Cell Origin The raster origin describes where within the cell the values are taken. Often when you are taking measurements the cell values of a raster, you will only take data from one position within each cell. That position is known as the cell origin. Other times, value contained within a cell will be the average value of the area the cell covers. In these cases the cell origin is less important

10 Raster Origin The raster s origin is where, geographically, the raster is located. It s what allows you to map the cell s values to a location in the real world. The rasters origin is taken relative to some position in the raster. Often that position is the lower left corner of the raster, but it varies depending on the format. Data Structures Continuous This is the most basic and straightforward raster representation. It is also the structure. Here, the value of each cell is sequentially written one right after the other. This structure has a flexibility advantage over other data structures since the values each cell can have aren t limited and is very simple to locate the value for a specific cell. However, it is not very space efficient without compression. Figure 14 Continuous Raster Figure 15 Classified version of figure 14 Classified With a classified data structure a map of possible cell values in the raster is first created. Then, the raster body is created with the actual cell values replaced by the index of those values in the map. Classified rasters are space efficient if there are few differing cell values. You also maintain the ability to quickly locate the value of a given cell. Web formats like PNG and GIF use classified rasters with compression since it allows them to put 24-bit or 32-bit images into 8-bit or 16-bit space

11 Compression Compression is used to reduce the final size of a raster by applying an algorithm to the raster that allows you to change the way that the data is represented. With rasters that are sufficiently large it is often vital that the raster be compressed. It can take a great deal more time load an uncompressed raster from a hard drive or the internet into memory than it takes to uncompress a raster already in memory. Compression algorithms are often divided into two types, lossy and lossless. With lossy compression algorithms the correctness of the data is often sacrificed in order to compress a raster in to as small a space as possible. Lossless algorithms, on the other hand, ensure that the data remains completely intact and because of this can t approach the compression ratios you can achieve with a lossy algorithm. The majority of efficient algorithms, both lossy and lossless, used these days require a strong mathematical background to understand, much less implement. Unfortunately, because of this, many widely used and important compression algorithms won t be included here. Below, a few relatively simple lossless compression algorithms are explained Run Length Encoding (RLE) Run Length Encoding is probably the most simple and naïve compression algorithm there is. With run length encoding, instead of specifying a value at every single location in the raster, you specify values as they appear and report how many occurrences of that value there are consecutively. RLE encoding is can save a great deal of space if many sequential cells contain the same values. LZW Compression LZW compression is a type of compression that allows you to optimize for space based on sequences of bytes occurring frequently. First a dictionary of the 255 possible 8-bit bytes is created and a code is assigned to each one. Then it traverses the binary file adding character sequences that it finds that are not already in the dictionary and assigns codes to each of those. The dictionary does not have to be transmitted with the compressed data since the decompressor can build it in the same way the compressor did. This compression algorithm is used by the GIF format

12 Case Study: ENVI.hdr format There are a number of features of the ENVI header format that make it a good introductory raster format. ENVI has a simple structure; there is no compression or other derivation from the standard continuous representation. This makes it easy to find individual cell values within the binary file. It also has a lot of flexibility in the kinds of data that can be stored. EVNI can have one of 6 data types for storing numerical data, it can also store color images in RGB format. There is even the option to specify geographic location information. An ENVI raster is divided into two separate files. One is a header file which stores all of the raster properties; such as its dimensions, data type and interleave. The second is the binary file which stores all of the actual raster data. Header File The header is a plain ASCII text file that has the.hdr extension. Each property of the raster is placed on a separate line in the format: [keyword] = [value] Below is a table of keywords and their possible values. ENVI.hdr Keyword samples lines bands interleave data type byte order (optional) map info (optional) Table 2 ENVI.hdr header file keywords Desciption Number of columns in raster Number of rows in raster Number of bands. If set to 3 or 4 most readers will interpret as RGB or RGBA color. Value can be one of bsq,bil or bip Possible values: 1 8-bit unsigned integer 2 16-bit signed integer 3 32-bit signed integer 4 32-bit real 5 64-bit real bit unsigned integer bit unsigned integer Possible values: 0 little endian (default) 1 big endian This keyword provides geographical positioning. See next page for complete description

13 Map Info Keyword: The map info keyword will have the following format: map info = {[projection], [reference cell x coordinate], [reference cell y coordinate], [cell easting], [cell northing], [x cell size], [y cell size], [projection zone], [direction]} [projection]: This is the projection system that the raster uses, will almost always be UTM in ENVI.hdr files. [reference pixel x/y coordinate]: These two fields specify the grid location of the first pixel in the binary file. This will be 1,1 for top-down vertical order or 1,<number of columns> for bottom-up. [pixel easting]: This is the number of meters east of the zone origin the reference pixel is located. [pixel northing]: This is the number of meters north of the projection zone origin the reference pixel is located. [x/y pixel size]: This is the size/resolution of each cell in meters. [projection zone]: This is the projection zone that the raster is located in. [direction](utm only): Because there are two of every UTM projection zone one on each side of the equator you must specify which one by entering North or South. Binary File After reading the header file you now know all of the raster s properties relevant to reading the binary image data. The ENVI.hdr format is a standard continuous format, so accessing individual cells of the raster is straightforward. Below, I ve included a table of formulae for accessing the individual cell values for a given row, column and band from the binary file based on the header properties. The table assumes that row, column and band are counted starting from zero. Interleave One band, no interleave BIP BIL BSQ Table 3 Formula for file position (row*samples + column)*allocation (row*samples + column)*allocation*bands+band (row*bands+band*column)*samples*allocation (lines*band*samples+row*samples + column)*allocation

14 References 1. Complete description of LZW Compression algorithm in Dr. Dobb s Journal by Mark Nelson. Created October Accessed January Reference in creating the ENVI.hdr Case study: RSI Documentation, ENVI Reader for ArcGIS User s Guide, March 2003, RSI Inc. 3. Reference for information about color spaces and color models. Unknown Author Updated January 11, 2006: Accessed January 11, Geographical raster information. All text and images in this document were created by Joseph Collins-Unruh