A Brief History of Pixel

Transcription

1 REPRINT author contact: A Brief History of Pixel Richard F. Lyon Foveon Inc., 2820 San Tomas Expressway, Santa Clara CA ABSTRACT The term pixel, for picture element, was first published in two different SPIE Proceedings in 1965, in articles by Fred C. Billingsley of Caltech s Jet Propulsion Laboratory. The alternative pel was published by William F. Schreiber of MIT in the Proceedings of the IEEE in Both pixel and pel were propagated within the image processing and video coding field for more than a decade before they appeared in textbooks in the late 1970s. Subsequently, pixel has become ubiquitous in the fields of computer graphics, displays, printers, scanners, cameras, and related technologies, with a variety of sometimes conflicting meanings. Reprint Paper EI Digital Photography II Invited Paper IS&T/SPIE Symposium on Electronic Imaging January 2006, San Jose, California, USA Copyright 2006 SPIE and IS&T. This paper has been published in Digital Photography II, IS&T/SPIE Symposium on Electronic Imaging, January 2006, San Jose, California, USA, and is made available as an electronic reprint with permission of SPIE and IS&T. One print or electronic copy may be made for personal use only. Systematic or multiple reproduction, distribution to multiple locations via electronic or other means, duplication of any material in this paper for a fee or for commercial purposes, or modification of the content of the paper are prohibited.

2 A Brief History of Pixel Richard F. Lyon Foveon Inc., 2820 San Tomas Expressway, Santa Clara CA ABSTRACT The term pixel, forpicture element, was first published in two different SPIE Proceedings in 1965, in articles by Fred C. Billingsley of Caltech s Jet Propulsion Laboratory. The alternative pel was published by William F. Schreiber of MIT in the Proceedings of the IEEE in Both pixel and pel were propagated within the image processing and video coding field for more than a decade before they appeared in textbooks in the late 1970s. Subsequently, pixel has become ubiquitous in the fields of computer graphics, displays, printers, scanners, cameras, and related technologies, with a variety of sometimes conflicting meanings. Keywords: Pixel, Bildpunkt, pel, pix, picture element, Billingsley, history, Foveon INTRODUCTION Long before Constantin Perskyi introduced the half-greek half-latin word television in France in 1900, Paul Nipkow had filed a German patent application on his mechanical-scanning TV or Elektrisches Teleskop 1 in 1884, in which he referred to Bildpunkte literally picture points but now universally translated as pixels. Actually, Bildpunkt had been used in photography before its use in TV scanning, in 1874 by Hermann Vogel, 2 as the point in the focal plane of a camera lens where rays from an object point converge. Arthur Korn, in his 1904 book Elektrische Fernphotographie, 3 clarified that in a scanning system a Punkt (point) ist streng genommen ein kleines Flächenelement... is strictly taken a small area element. Germans make up compound nouns routinely: the one-third-german two-thirds-greek compound Fernphotographie for distance photography is not unusual, and neither was Bildpunkt nor Flächenelement. The incorporation of such terms as picture point, area element, andpicture element into English as informal two-word compounds is equivalently unremarkable, which is why the terminology had a hard time converging until the portmanteau pixel was introduced much later. We review the history of picture element and related terms below, concentrating on the origins of pixel in the 1960s, and its gradual popularization in the 1970s and 1980s. PICTURE ELEMENT An appreciation of the origins of pixel demands some understanding of the origins and meanings of picture element. It was introduced in Wireless World magazine in 1927, in a long news item Television Demonstration in America by Alfred Dinsdale; 4 see Figure 1. Dinsdale had written the very first English book on Television in 1926, 5 but instead of picture element he had used there lots of other colorful language: a mosaic of selenium cells, a great number of small parts, thousands of little squares, and a succession of little areas of varying brilliance. Figure 1. The first appearance of picture element, in a news item in Wireless World and Radio Review, about a demonstration by Ives at Bell Labs of a 50-by-50-element television system.

3 Subsequently, picture element appeared in books on television by H. Horton Sheldon and Edgar Norman Grisewood in and by A. Frederick Collins in It was used by RCA researcher Alfred N. Goldsmith in (along with dot-elements), followed in the 1930s by numerous papers by Vladimir K. Zworykin and a dozen other RCA authors. But the use of picture element died out as quickly as it had started after Zworykin wrote in 1937, The picture element is a purely fictitious concept when applied to the mosaic. 9 A few RCA researchers, notably Albert Rose 10 and Otto Schade, 11 continued to use picture element to examine the theory of imaging, but with differing interpretations. Rose wrote in 1946, A picture element is here taken to be an element of area of arbitrary size, not necessarily the smallest resolvable area. Schade wrote in 1948, The smallest detail which can be resolved by an imaging process will be defined as a picture element. This dual meaning, between an arbitrary element and a resolution element, persists even today with pixel. Picture element was picked up by Don Fink in his 1940, , 13 and 1957 books 14 on Television Engineering, following Schade s interpretation, but to a large extent the concept was just ignored, since television scan lines did not have discrete elements, and since resolution was routinely quoted in TV Lines. Bell Labs authors used image elements and other terms, with the exception of John R. Pierce who used picture element in his 1956 book Electrons, Waves, and Messages. 15 Pierce, who coined the term transistor, introduced lay readers to modern physics and engineering concepts, including the use of a Nipkow disk for television scanning; see Figure 2. His 1945 patent application 16 was written to include picture transmission, and has code elements, code groups, andsamples, but no picture elements. Figure 2. Picture element was never used by Bell Labs authors until John R. Pierce used it to explain television via the Nipkow disk in his 1956 book Electrons, Waves, and Messages. As computerized scanning, processing, and reproduction of images were being explored in the 1950s 17,18 and 1960s, picture element was seldom used in these fields, with a few exceptions such as Schreiber, 19 Roberts, 20 and Seyler. 21 For example, in the 1968 collection Pictorial Pattern Recognition, 22 we find resolution elements, positions, spots, sample spots, samples, gray values, raster points, matrix elements, video element, point, digital sample, beam spot, digitalpicture element, and, yes, even picture element. These don t show any more consistency than terms used in the 1920s, which included image element, area element, elementary area, picture units, small squares, little parts, units, dots, points, discrete signals, portions, elemental area of the picture, and elemental tone value.

4 FINDING PIXEL According to the Oxford English Dictionary, 23 pixel dates from the 1969 article Mariner 6 Television Pictures: First Report by Caltech and JPL authors in Science magazine. 24 That paper associates pixel with both picture element and resolution element; see Figure 3. I found this unsatisfying as a source, so I kept looking. The only one of the authors I could contact, atmospheric scientist Andrew T. Young, said that he was mildly horrified that the OED had associated him with pixel and said, I thought it was a vile neologism, and tried to avoid it myself. It was in use by the guys in the Image Processing Lab. He wanted to use the term sample but that had statistical meanings, so it was a problem for his co-authors. Figure 3. The 1969 table in Science magazine 24 that associates pixels with resolution elements. Through Jerry Solomon of Caltech, an ex-head of the JPL Image Processing Lab (IPL), I found out in 2002 from Fred Billingsley, retired from the IPL, that he recalled pixel coming from a subcontractor, probably the Link Division of General Precision in Palo Alto, by 1964 or earlier. Shortly thereafter, Fred died, leaving me with only that clue. Shortening the story by omitting a few years of dead ends, frustration, and delays, I finally got from Keith E. McFarland, a Link ex, out of his barn in Oregon, manuals for the Video Film Converter (VFC) that he had built at Link for JPL in the mid 1960s. Both the 1966 and 1967 manuals 25,26 used the term pixel in various ways, without defining it. McFarland says that pixel was in use at the time, and that they did not invent it at Link. He also confirmed for me that the pizel I had found in a 1968 engineering case study 27 on his VFC project was just a typo. PIXEL IN SPIE ELECTRONIC IMAGING It took a while more, but I eventually learned of and obtained copies of papers that Fred Billingsley had presented at two different SPIE meetings in 1965, and found that both used pixel. The first of these, entitled Digital Video Processing at JPL, 28 is in SPIE Vol. 3, the third annual SPIE Seminars in Depth, entitled Electronic Imaging Techniques for Engineering, Laboratory, Astronomical, & other Scientific Measurements, whichwasheldinlos Angeles in April, Eugene Turner of the Aerospace Corporation writes in the Foreword: Response to this seminar by both the authors and the audience has borne out the wisdom of the choice of Electronic Imaging. The present meeting would appear to be a direct descendent of this one at which pixel was first published. Furthermore, both papers 28,29 included a photo of Fred and his technician standing with the Video Film Converter that they had recently acquired from Link. Figure 4 is a copy of that photo, and Figure 5 shows the text fragment in which pixel was first used in print.

5 Figure 4. My work on the history of pixel is dedicated to the memory of Fred Crockett Billingsley ( ). Here he is shown, behind the oscilloscope, in the Image Processing Lab at JPL with technician George Peterson and the Link Video Film Converter, in a photo that was included in both of Billingsley s 1965 SPIE papers. Figure 5. Pixel s first known appearance in print already has a confused meaning. If a pixel is a sample, why do we have to sample each pixel? I m still looking for more, but besides these papers by Billingsley and his associates, and a 1969 JPL paper on TV data compressionbyrobertrice, 30 and another article in Science on the Mariner Mars pictures, 31 I find no other uses of the word pixel in the 1960s. Anyone who knows anything about Link or the early use of such terms is invited to contact me. About the same time, pel was introduced as another abbreviation for picture element. This one has a more much definite origin, in 1964, when MIT Professor William F. Schreiber asked his secretary Claire Kay to make him a good abbreviation to use instead of the German BP for Bildpunkt that was then sometimes used. Dr. Schreiber has clear memories of the names and dates, set off by his time in India, after which he published a paper using pel in the Proceedings of the IEEE in Nonetheless, he was not so successful at getting the term accepted, even by his coauthors such as Troxel and Seitz when they did a paper together in 1969 on a wirephoto converter and used simply element. 33

6 PIX FOR PICTURES The Dictionary of American Slang 34 explains that pix was popularized for movies (picture shows) by Variety magazine headline writers before 1936, and has been in wide newspaper use for photographs since about Based on the existence of the Australian photo magazine Pix, started in 1938, the use of pix for photos may be older than they realized, at least down under. See Figure 6. Figure 6. The use of ix for ics appeared in the pics biz by at least 1921, in Paramount s Publix Theatres. Variety used pix in headlines at least by 1934; on 27 March we find two uses of PIC and two of PIX on one page; on 24 April SEES MORE PIX THAN ANY PREZ before the famous 1935 STICKS NIX HICK PIX. After the Australian magazine PIX started in 1938, the term pix became commonplace in photojournalism. Apparently The New Costume includes a cigarette for the lady. Fred Billingsley used pix in a table of his 1970 paper on applications of digital image processing, 35 under a use heading, with the phrases convert slant pix to ground projection and overlay match of two pix. Perhaps he was thinking that by using pix he would help pixel become more acceptable. PIXEL ESCAPES INTO THE WILD Peter M. Will et al. usedpixel in an IBM internal report in , and in a remote-sensing workshop version of the same paper in These, and a 1971 IBM patent application by Will et al., 38 are the first known uses of pixel outside of JPL and Link. Will recalls that he was working on image processing, remote sensing, and DPCM image coding, using pel, when an IBM colleague told him that was old-fashioned and he should use pixel instead. He switched, but IBM mostly stayed with pel. I have found few other uses of pixel through JPL author Tom Rindfleisch published pixel in a graph axis label in a 1970 workshop 39 and a 1971 paper; 40 see Figure 7. In the same workshop, JPL author G. Edward Danielson, Jr. uses pixels/line and Quantization/pixel in a table. 41 JPL author Robert Rice used pixel in an internal JPL memo in 1969, 30 and published with James Plaunt the first IEEE journal article with pixel in NASA introduced pixels to the general public in the 1971 book Mariner 6 & 7 Pictures of Mars. 43 Lynn Quam used pixel in his 1971 Stanford Ph. D. dissertation 44 on comparing Mars images to look for signs of life; Rindfleisch helped with that project and went from JPL to Stanford about that time, taking pixel with him.

7 Figure 7. The earliest places where pixel escaped into the wild tended to be hidden from picky editors in graphs, captions, and table entries. Rindfleisch s 1970 CYCLES/PIXEL and Pratt, Chen, and Welch s bits/pixel are two examples. PIXEL GROWS UP The 1970s were a period of exponential growth for pixel. Starting with the 1971 IBM filing mentioned above, U. S. patent applications using pixel doubled almost annually through the 1970s (and perhaps longer, but it gets hard to count); see Figure 8. Patents with pel also doubled annually, but in numbers lagging the pixel applications by about two years; the overwhelming majority of pel patents were assigned to IBM and Bell Labs. Figure 8. Moore s law? The graph shows the approximate annual doubling of U. S. patent applications using the words pixel and pel during the 1970s (issued patents are counted in their year of filing). The searchable databases of the IEEE, the ACM, the USPTO, and others, provide a pretty good coverage of usage since 1970, but not before. I can say a lot about how pixel spread in the early 1970s, based on tracking people and papers, e.g. from JPL to Stanford via Tom Rindfleisch; from JPL to USC via William Pratt; from USC to SUNY and then to UC Davis via Anil K. Jain; from Stanford to CMU via Raj Reddy. Many of the people are still around to provide information and documents, and to confirm hunches. One thing that becomes clear is that it was the image processing and artificial intelligence community that propagated the term pixel, not the computer graphics community as Nicholas Negroponte states in his 1995 book Being Digital. 46 For example, none of the Utah computer graphics people used the term pixel until they had moved on from Utah, in the

8 late 1970s. The first uses of pixel in Utah dissertations were by Tom Stockham s students doing image processing, starting with Olivier Faugeras in 1976 who demonstrated color images coded at an average bit rate of 1 bit/pixel. 47 Nobody at Xerox PARC, NYIT, or other leading computer-graphics institutions had adopted pixel by then. The first use of pixel at Xerox was in a 1976 patent application by Michael Wilmer on facsimile image compression and OCR, 48 which said picture cells or pixels, even while the nearby pixels of Alto computers, SuperPaint screens, laser printers, and Smalltalk/Bitblt graphics were called something else. One reason for this disconnect that I discovered: the graphics community had a different meaning for picture element, namely, a graphical element such as a line, circle, polygon, character, etc. To them, abbreviating picture element as a name for their raster element or sample just didn t fit. In 1975, Ed Catmull 49 illustrated some of the terminology of the graphics field in a diagram reproduced in Figure 9. Figure 9. In 1975, Ed Catmull laid out the terminology used in the computer graphics field, covering a number of concepts that today all go under the term pixel; there is no picture element among them, just raster elements, dots, raster-element squares, and intensity values. The earliest publications of pixel outside of JPL authors were in 1971, as mentioned above. Several more academic, theoretical, and medical image processing papers appeared in 1972 and Pixel was widely propagated in the late 1970s through textbooks by Raphael Gonzalez and Paul Wintz (1977), 50 William Pratt (1978), 51 and Kenneth Castleman (1979); 52 all three books were titled Digital Image Processing. PIXEL IN TITLES Once pixel reached a critical recognition level with the help of the textbooks, it was widely adopted in the 1980s. The first paper with pixel in its title appeared in When Adele Goldberg and Robert Flegal of Xerox PARC wrote the column Pixel Art, 53 they still felt a need to put pixel art in quotation marks and pixel in italics, and to explain about

9 a bitmap which indicates the black and white cells or pixels of the image being represented. See Figure 10. Like Mike Wilmer at Xerox before them, these authors seem to take pixel to represent pic cell or picture cell as opposed to picture element. Figure 10. Goldberg and Flegal were the first to use pixel in a title, in this Pixel Art column in This clipping is here dedicated to the memory of Joseph Maleson, who worked on the error-diffusion dithering used in these images and built the scanner used to capture them. By the late 1980s, pixel was appearing in book titles. The first was probably JeffreyYoung s Inside MacPaint: Sailing Through the Sea of Fatbits on a Single-Pixel Raft 54 in By 1990, we find a magazine with pixel in the title Pixel: The Magazine of Scientific Visualization. 55 The more modern history of pixel involves computer graphics, monitors, printers, image processing, image and video compression, image scanners, digital cameras, the internet, and all related products and technologies. Even the German children s book Alexandra und der Pixel 56 uses it in 2000 in preference to Bildpunkt; see Figure 11. And it is used in many other languages, too; see Figure 12. Figure 11. Pixel has migrated to German, in the title of this children s book published in 2000.

10 Figure 12. Pixel has migrated to French, a language that tries to resist the temptation to incorporate foreign words. Photo by Andrew Piner. PIXELS IN SENSORS The television and CCD image-sensor community also began to incorporate pixel into their language in the 1970s and 1980s. The unit cell of a solid-state image sensor was first called a picture element by P. K. Weimer et al. ofrcain 1969; 57 the first to call the unit cell of a CCD a picture element may have been Yamanaka of Sony in In 1980, at Xerox PARC, I called the unit cell of my own optical mouse sensor a pixel. 59 Mostly, though, through the 1970s, picture element and pixel were used for the information elements measured by such cells, not for the sensor cells themselves. For example, a 1974 CCD paper by Dyck and Jack 60 of Fairchild consistently refers to the array units as photoelements or elements, and uses photons/pixel only in quantifying light signal levels. Figure 13. Lampe and White s 1975 pixel pair referring to an element of detector hardware.

11 The term pixel pair was used for a hardware sensor element in a 1975 patent application by Lampe and White 61 of Westinghouse, as shown in Figure 13, though they didn t use pixel for a hardware element in their 1974 or later papers, where they use it for an information element. As far as I can tell, this patent application is the first use of pixel to designate an element of sensor hardware. CAMERA PIXELS BY THE MILLIONS Most consumer digital cameras use an image sensor that is covered by a mosaic of color filters, one filter over each sensor element, in a pattern disclosed by Bryce Bayer of Eastman Kodak Company in a 1975 patent application. 62 The Bayer pattern uses 50% green filters, and 25% each of red and blue. The Bayer patent refers to luminance elements (for the green ones) and chrominance elements (for the red and blue ones), without calling them picture elements, asseenin Figure 14. Figure 14. Bryce Bayer s 1976 patent application refers to luminance-sensitive elements and chrominance-sensitive elements, not picture elements, nor pixels. The term CFA (color filter array) and another CFA pattern with the same proportions as Bayer s were published at a 1976 IEDM meeting by other Kodak authors Dillon et al.; 63 but the pattern is different, optimized for interlaced video readout. Their paper uses the term picture elements in reference to these units of imager hardware at each cell, following what Weimer did in 1969 but applying it for the first time to the single-color-sensing elements of a color imager, in anticipation of the modern trend of calling such things pixels. In 1982, Aoki et al. 64 of Hitachi applied the term pixel to the individual single-color elements of a CFA color image sensor. Berger et al. repeated this twist in their 1984 ISSCC paper with pixel in the title, A line transfer color image sensor with 576x462 pixels. 65 Thus began the use, prevalent today, of the term pixel for a sensor element that measures only one-third of the information needed to make a color (RGB) pixel. It is convenient that the single-color-measuring element in a CFA (e.g. Bayer pattern) CCD or CMOS image sensor is in one-to-one correspondence with an RGB image pixel interpolated from the measured image samples through a process known as demosaicking. Because of this coincidence, it has been easy to label sensors and cameras by the number of the pixel sensors in them, and have that match the numbers of pixels in the output image files.

12 This convenient situation was shattered, however, when Fujifilm introduced the SuperCCD arrangement of pixel sensors, 66 at alternate locations on a square grid, like the black squares on a checkerboard (alternatively, think of it as a 45-degree-rotated square grid). The natural interpolated file size then had twice as many RGB pixels as the image sensor had pixel sensors, resulting in considerable confusion in the market when they initially promoted the higher output file pixel counts. Under pressure from the industry, Fujifilm began to label their cameras primarily with the number of actual pixel sensors instead, firming up the de facto definition of camera pixels as single-color sensor elements independent of the number of output pixels. A similar confusion arose at the introduction of the Foveon X3 sensor technology in In Foveon s sensors, the sensor elements are arranged in a 3D grid, in layers that take advantage of the wavelength-dependent absorption coefficient of light in silicon; see Figure 15. Dick Merrill s patent 68 states, Three sets of active pixel sensor circuitry are coupled to the three detector layers, such that three active pixel sensors are formed using the group of three colocated detectors of the vertical color filter detector group. The natural file size for capturing all the information from these sensor elements has only one RGB pixel per stack of three pixel sensors. For example, the Sigma SD9 and SD10 cameras produce files of 3.4 M RGB pixels from Foveon X3 sensors having 10.2 M elements in three layers of 3.4 M per layer. Unlike the natural output file sizes of the CFA sensors, the Foveon s compact output file size involves no interpolation. Foveon and Sigma tried to bridge the confusion by referring to the Sigma SD9 camera as 3.4 megapixels x 3; but many retailers reduced it to simply 3.4 MP, or put it in a 3 MP category, a complete mischaracterization relative to all other digital cameras, and not in alignment with the precedents of counting actual pixel sensor elements. Figure 15. A Foveon patent drawing showing how three stacked pixel-sensor photodiodes may be built in multiple epitaxial silicon layers using multiple ion implants. As with the Fujifilm technology, the Foveon technology leads to the need to distinguish between the resolution, the number of pixels in the output file, and the number of pixel sensors in the image sensor. However, popular usage has associated resolution with pixel count, in files and cameras and even in the 1969 paper on Mars images; and the Foveon technology has a somewhat different relationship of pixel sensor counts to resolution than the various CFA technologies have. Therefore, using Foveon s higher total number of pixel sensors as a camera s megapixel (MP) rating, in agreement with industry practice and guidelines, 69 has not been universally understood or welcomed. See Figure 16. How do pixel counts relate to resolution, in historical usage? All of the old television literature, and most of the current definitions, analyses, and textbook uses of pixel as a resolution element concern monochrome systems only. The number of pixels in an image is the number of image samples, and provides a useful correlate to a bound on image resolution. Product ratings in megapixels for displays, scanners, and monochrome cameras are often in agreement with these historical uses in that the luminance component at least has a resolution corresponding to the equivalent monochrome image of the same number of pixels. For color image sensors and cameras, however, the connection between resolution and pixel count was broken when individual single-color-sensitive pixel sensors began to be counted as pixels, even before the advent of digital cameras. So color cameras have their own, different, somtimes inconsistent, relationship between resolution and pixel count.

13 Since the typical Bayer-pattern imager uses 50% of its pixel sensors for the green or luminance channel, and 25% for each of two chroma dimensions (red and blue), the resolution of the luminance component is consistent with a monochrome sensor with 50% as many pixels. The chrominance component has a resolution consistent with the resolution limit of a file of 25% as many RGB pixels, at best; typical demosaicking algorithms make tradeoffs that make the chrominance resolution even worse than this, in an attempt to reduce color aliasing artifacts. 70 The Foveon X3 sensor, on the other hand, has both the luminance and chrominance components uniformly sampled, consistent with the resolution of a monochrome sensor or a file of 33.3% as many RGB pixels as the number of sensor elements. One suggestion has been to introduce a Bayer-equivalent pixel count that lines up the luminance resolutions of the sensors, ignoring the chrominance. By such a measure, the Sigma camera might be rated about 6.8 MP Bayerequivalent. This approach, however, has no historical precedent, and under-represents the quality advantage of the X3 sensor s large number of elements by ignoring the advantage in chrominance resolution and reduced aliasing. In the camera field, the precedents, standards, guidelines, and current practice all point to simply counting sensor elements as the pixel count of a camera, and separately counting file pixels as a specification of output file size, even when they are not the same. Additional information about the organization of pixels is appropriate, especially when it is not the default Bayer CFA. It is unfortunate that the history of the camera industry has enshrined the historical confusion about pixels into such a prominently visible specification, but such is history. Figure 16. Our What s in a Megapixel slide ( and pixelpage-d.html), which attempted to show the relationships between different meanings of pixel, was not easy to translate to other languages for our web site. Auf Deutsch, everything kept coming out Bildpunkt, destroying the distinctions we were trying to draw; finally, we got the translators to agree to this version. It is not possible for a small company such as Foveon to succeed in straightening out the confusion, as we tried when initially referring to our sensors as 3.4 MP x 3. We can not tolerate having our hi-end sensors referred to simply as 3.4 MP, or put into a 3 MP category, so we try to live with the confusion by adopting the usual convention plus adding clarifiers to total pixel numbers whenever possible. For example, we describe the sensors in the Sigma SD10 camera as 10.2 MP (3.4 MP Red MP Green MP Blue) so that nobody will wonder if we meant 10.2 M of each color. When we compare with competitive cameras, we clarify in the same way; for example 8 MP (2 MP Red + 4 MP Green + 2 MP Blue) to help people understand that a higher green or luminance resolution comes with a lower chrominance resolution.

14 CONCLUSION History continues to be made. Even in display specifications, there is a recent trend with small LCD display panels to copy the color camera convention and count each individual red, green, or blue element as a pixel instead of calling them sub-pixels like others in the industry do. I recommend fighting that trend, based on all the trouble that was caused by referring to single-color image sensor elements in cameras as pixels starting way back in My first claims are all based on only what I could find. As Tom Lehrer says about elements, There may be many others but they haven t been discovered. 71 ACKNOWLEDGEMENTS My old friends and renewed acquaintances who cooperated with my questioning over the last four years get my heartfelt thanks. In approximate order of contact, they are: Peggy Asprey, Carver Mead, Ivan Sutherland, John Pierce, Frank Crow, Lynn Quam, Keith Nishihara, Robert Gray, Ralph Algazi, Dick Duda, Henry Fuchs, Peter Will, Daniel Lau, Bob Sproull, Forest Baskett, Jim Morris, Al Kossow, Carlo Séquin, Jim Kajiya, Dick Shoup, Daniel Levitin, Martin Newell, Raj Reddy, Steve Rubin, Lance Williams, Al Davis, John Warnock, Dan Ingalls, Ted Kaehler, Alan Kay, Chuck Seitz, Al Oppenheim, Brian Barsky, Al Alcorn, Peter Hart, Patrick Baudelaire, Sandy Pentland, Lynn Conway, Dick Sites, Eugene Miya, Russel Martin, and Paul Hubel. My special appreciation goes to all the new friends I've made, people who in many cases responded to s or phone calls out of the blue: Tom Rindfleisch, Tom Binford, Azriel Rosenfeld, Bob Deen, William Schreiber, Barry Haskell, Fabian Pease, John Limb, Tom Huang, Jerry Solomon, Ken Castleman, Bob Selzer, Bill Pratt, Geza Kardos, Larry Roberts, Peter Tischer, Keith McFarland, Gordon Leckenby, Steve Gass, Kelly Heaton, George Dyson, Peter Millett, Tom Genova, Yun-Qing Shi, Andrew Young, John Thorne, Justin Rennilson, Rodger Carter, Gifford Hitz, Peter Retzinger, Ken Smith, Yuen Chung Kwong, Robert Rice, Ed Thelen, Lloyd Welch, Wen-Hsiung Chen, Alexander Sawchuck, Gabor Herman, Leonard Kleinrock, Elizabeth Rye, Charles Avis, Don McLean, Bruce Damer, Jeff Meyer, William von Hagen, Samuel Dwyer III, Rick Rashid, Doug Harper, Alvy Ray Smith, Andries van Dam, Ralph Guggenheim, Ed Catmull, Les Earnest, Dave Kasik, Len Shustek, Steven Gabriel, Elliott Levinthal, Robert Tucker, Bruce Baumgart, Irwin Sobel, Raphael Gonzalez, Jae Lim, Bob Shipley, Gwilym Lodwick, Joseph Newcomer, David W. Knight, Robert Vincent, Ernest Hall, Cliff Reader, Russell Kirsch, Raphi Rom, Sid Owen, Fred Brooks, and Evelyn Billingsley. REFERENCES 1. Paul Nipkow, Elektrisches Teleskop, German Patent No , Jan Hermann Vogel, Die chemischen Wirkungen des Lichts und die Photographie in ihrer Anwendung in Kunst, Wissenschaft und Industrie, Leipzig: F. A. Brockhaus, Arthur Korn, Elektrische Fernphotographie und Ähnliches, Leipzig: Verl. v. S. Hirzel, A. Dinsdale, Television Demonstration in America: A Successful Public Demonstration of Television between Washington and New York, Wireless World and Radio Review, Vol. XX, pp , June 1, Alfred Dinsdale, Television, London: Pitman, H. Horton Sheldon and Edgar Norman Grisewood, Television, New York: Van Nostrand, A. Frederick Collins, Experimental Television: How to Make a Complete Home Television Transmitter and Receiver, Boston: Lothrop, Lee & Shepard, 1932; reprinted Bradley IL: Lindsay Publications, Alfred N. Goldsmith, Image Transmission by Radio Waves, Proc. I.R.E., Sept. 1929; reprinted in Radio Facsimile, pp , New York: RCA Institutes Technical Press, V. K. Zworykin, G. A. Morton, and L. E. Flory, Theory and Performance of the Iconoscope, Proc.I.R.E., Aug., 1937; reprinted in Television Vol. II, New York: RCA, Albert Rose, A Unified Approach to the Performance of Photographic Film, Television Pickup Tubes, and the Human Eye, J. Soc. Motion Picture Eng., Oct., 1946; reprinted in Television Vol. IV, pp , Princeton: RCA,1947.

15 11. Otto H. Schade, Electro-Optical Characteristics of Television System, RCA Review, Sept. 1948; reprinted in Television Vol. V, pp , Princeton: RCA, Donald G. Fink, Principles of Television Engineering, New York: McGraw Hill, Donald G. Fink, Television Engineering, 2nd ed., New York: McGraw Hill, Donald G. Fink, ed., Television Engineering Handbook, New York: McGraw Hill, John R. Pierce, Electrons, Waves, and Messages, Garden City: Hanover House, John R. Pierce, Communication System Employing Pulse Code Modulation, U. S. Patent 2,437,707, Mar (filed 1945). 17. R. A. Kirsch, L. Cahn, C. Ray, and G. H. Urban, Experiments in Processing Pictorial Information with a Digital Computer, Proc. Eastern Joint Computer Conf., pp , New York: I.R.E., W. M. Goodall, Television by Pulse Code Modulation, Bell Sys. Tech. J., Vol. 30 pp , W. R. Schreiber, The Measurement of Third Order Probability Distributions of Television Signals, IRE Trans IT, Vol 2, pp , Lawrence G. Roberts, Picture Coding Using Pseudo-Random Noise, IRE Trans. IT, Vol. 8, pp , A. J. Seyler, The Coding of Visual Signals to Reduce Channel-Capacity Requirements, Proc. IEE, Part C, Vol. 109, pp , George C. Cheng, Robert S. Ledley, Donald K. Pollock, and Azriel Rosenfeld, Pictorial Picture Recognition, Washington: Thompson Book Co., Oxford English Dictionary, online edition, oed.com, Robert B. Leighton, Norman H. Horowitz, Alan G. Herriman, Andrew T. Young, Bradford A. Smith, Merton E. Davies, Conway B. Leovy, Mariner 6 Television Pictures: First Report, Science 165, pp , 15 Aug Utility Manual for the Media Conversion Film Recorder, WDIM-121, Link Group of General Precision, Inc., Systems Division, Palo Alto, Utility Manual for Video Film Converter, WDIM 138, Link Group of General Precision, Inc., Systems Division, Palo Alto, Charles Fernald, Keith McFarland: Removing Pincushion Distortion in a Cathode Ray Tube, Engineering Case Library ECL-112A, Stanford, F. C. Billingsley, Digital Video Processing at JPL, in Eugene B. Turner (ed.), Electronic Imaging Techniques I, Proceedings of SPIE, Vol. 0003, pp. XV-1 19, 1965 (Apr. 1965, Los Angeles; publication date listed in error in SPIE catalog as Jan. 1964). 29. Fred C. Billingsley, Processing Ranger and Mariner Photography, in Computerized Imaging Techniques, Proceedings of SPIE, Vol. 0010, pp. XV-1 19, Jan (Aug. 1965, San Francisco). 30. Robert F. Rice, The Code Word Wiggle: TV Data Compression, JPL Tech. Memo , Pasadena: NASA, R. B. Leighton, N. H. Horowitz, B. C. Murray, R. P. Sharp, A. G. Herriman, A. T. Young, B. A. Smith, M. E. Davies, C. B. Leovy, Mariner 6 and 7 Television Pictures: Preliminary Analysis, Science 166, pp , 3 Oct William F. Schreiber, Picture Coding, Proc. IEEE, Vol. 55, Mar Donald E. Troxel, William F. Schreiber, and Charles L Seitz, Wirephoto Standards Converter, IEEE Trans. Comm. Tech., Vol. COM-17, pp , Oct Harold Wentworth and Stuart B. Flexner (eds.), Dictionary of American Slang, New York: Thomas Y. Crowell Co., F. C. Billingsley, Applications of Digital Image Processing, Applied Optics, Vol. 9, pp , Feb P. M. Will, R. Bakis, and M. A. Wesley, On an All-Digital Approach to Earth Resources Satellite Image Processing, IBM internal memo RC 3027, Sept R. Bakis, M. A. Wesley, and P. M. Will, Digital Correction of Geometric and Radiometric Errors in ERTS Data, Proc. of the Seventh Intl. Symp. on Remote Sensing of Environment, Vol. II, pp , Willow Run Laboratories, Ann Arbor, Michigan, Peter A. Franaszek, David D. Grossman, and Peter M. Will, Differential Encoding with Lookahead Feature, U. S. Patent 3,720,875, Mar. 13, 1973 (filed 1971). 39. Thomas C. Rindfleisch, Digital Image Processing for the Rectification of Television Camera Distortions, NASA SP-256: Symp. Astronomical Use of Television-Type Image Sensors, pp , Princeton, May, T. C. Rindfleisch, J. A. Dunne, H. J. Frieden, W. D. Stromberg, and R. M. Ruiz, Digital Processing of the Mariner 6and7Pictures, J. Geophysical Res., Vol. 76, pp , Jan

16 41. G. Edward Danielson, Jr., Experience from Mariner TV Experiments, NASA SP-256: Symp. Astronomical Use of Television-Type Image Sensors, pp. 1 16, Princeton, May, Robert F. Rice and James R. Plaunt, Adaptive Variable-Length Coding for Efficient Compression of Spacecraft Television Data, IEEE Trans. Comm. Tech., Vol. COM-19, pp , Dec Stewart A. Collins, NASA SP-263: The Mariner 6 and 7 Pictures of Mars, Pasadena: NASA, Lynn H. Quam, Computer Comparison of Pictures, Stanford Artificial Intelligence Laboratory Memo AIM-144, STAN-CS , Stanford, William K. Pratt, Lloyd R. Welch, and Wen-Hsiung Chen, Slant Transforms for Image Coding, Proc. Symp. On Appl. Of Walsh Functions, Mar Nicholas Negroponte, Being Digital, New York: Knopf, Olivier D. Faugeras, Digital Color Image Processing and Psychophysics within the Framework of a Human Visual Model, Ph.D. dissertation, Univ. Utah, Salt Lake City, June Michael E. Wilmer, Optical Character Recognition System, U. S. Patent 4,034,343, July 1977 (filed 1976). 49. Edwin Catmull, Computer Display of Curved Surfaces, Proc. IEEE Conf. on Computer Graphics, Pattern Recognition, and Data Structures, Los Angeles, May 1975; reprinted in Rosalee Wolfe (ed.), Seminal Graphics: Pioneering Efforts that Shaped the Field, ACM, Raphael Gonzalez and Paul Wintz, Digital Image Processing, Reading: Addison Wesley, William K. Pratt, Digital Image Processing, New York: Wiley Interscience, Kenneth R. Castleman, Digital Image Processing, Englewood Cliffs: Prentice Hall, Adele Goldberg and Robert Flegal, ACM President's Letter: Pixel Art, Commun. Assoc. Computing Machinery, Vol. 25, pp , Jeffrey S. Young, Inside MacPaint: Sailing Through the Sea of Fatbits on a Single-Pixel Raft, Bellevue: Microsoft Press, Pixel: The Magazine of Scientific Visualization, Vol. 1, Watsonville CA: Pixel Communications, Inc., Adrian Tobler, Alexandra und der Pixel, Basel: Christoph-Merian-Verl., P. K. Weimer, W. S. Pike, G. Sadasiv, F. V. Shallcross, and L. Meray-Horvath, Multielement Self-Scanned Mosaic Sensors, IEEE Spectrum, Vol. 6, No. 3, pp , Mar Seisuke Yamanaka, Yasuo Kanou, Tadayoshi Mifune, and Satoshi Shimada, Solid state camera, U. S. Patent 3,975,760, Aug (filed 1975). 59. Richard F. Lyon, The Optical Mouse, and an Architectural Methodology for Smart Digital Sensors, in Kung, Sproull, and Steele (eds.), CMU Conference on VLSI Structures and Computations, Pittsburgh: Computer Science Press, October R. H. Dyck and M. D. Jack, Low Light Level Performance of a Charge-Coupled Area Imaging Device, Proc. Intl. Conf. Tech. And Appl. of Charge-Coupled Devices, Edinburgh, Sept Donald R. Lampe and Marvin H. White, CCD Focal Plane Processor for Moving Target Imaging, U. S. Patent 4,064,533, Dec (filed 1975). 62. Bryce E. Bayer, Color Imaging Array, U. S. Patent 3,971,065, July 1976 (filed 1975). 63. P. L. P. Dillon, A. T. Brault, J. R. Horak, E. Garcia, T. W. Martin, and W. A. Light, Integral Color Filter Arrays for Solid-State Imagers, Tech. Dig. Int. Electron Device Meeting, Washington, Dec Masakazu Aoki, Haruhisa Ando, Shinya Ohba, Iwao Takemoto, Shusaku Nagahara, Toshio Nakano, Masaharu Kubo, and Tsutomu Fujita, 2/3-Inch Format MOS Single-Chip Color Imager, IEEE J. Solid-State Circuits, Vol. SD-17, pp., , Apr Jean L. Berger, Jouis Brissot, Yvon Cazaux, Pierric Descure, A Line Transfer Color Image Sensor with 576x462 Pixels, Intl. Solid-State Circuits Conf., pp , Feb Dave Etchells, Fuji Finepix 4700, The Imaging Resource, Oct Mike Chambers, A View from the Crow s Nest: The Future of Digital Photography, PC Update Online The Melbourne PC Users Group, Richard B. Merrill, Vertical color filter detector group and array, U. S. Patent 6,632,701, Oct Guideline for Noting Digital Camera Specifications in Catalogs (JCIA GLA03), pdf/jcia_gla03english.pdf, Tokyo: Camera & Imaging Products Association, Paul M. Hubel, John Liu, and Rudolph J. Guttosch, Spatial Frequency Response of Color Image Sensors: Bayer Color Filters and Foveon X3, Santa Clara: Foveon, Tom Lehrer, The Elements, (song; see