Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers

Size: px
Start display at page:

Download "Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers"

Transcription

1 Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers (DOI: / ) This is a posting of a bibliography on gesture user interfaces, on-line character recognition (a.k.a. dynamic character recognition, a.k.a. pen computing), and related topics, including both hardware and software. I am posting it as a service to those with interest in the field. It may also be of special interest to anyone investigating any of the flood of patents in the areas of digitizer tablets, character recognition, tablet-pc GUIs, and multi-touch computing (such as the iphone and Touch ipod). It covers the time period from approximately 1891 (first electronic tablet) through 1914 (first handwriting-recognition input system) to the present day. This compilation is copyright Jean Renard Ward, 1992, 1996, 2003, 2005, 2008, 2009, 2011, 2013, Permission is hereby given to link to this list, or to cite it provided that notice of the source is given as stated below, that the full URL of this link is included, and also that you use my correct full name "Jean Renard Ward" is included with any such reference or link. Permission is hereby given to reproduce any diagrams or photographs in this collection for any purpose (to the extent that the materials are not subject to copyright by persons other than myself), including for the purposes of confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: "Source: Annotated Bibliography in Touch/Pen Computing and Handwriting Recognition by Jean Renard Ward". The bibliography is broken up into several web pages, for convenience in printing: 1891 to to to to to to to to to to to to to to to to to to 2015 not yet catalogued Pictures and Videos When was Pen Computing invented? Check this out: Notes on the "unknown" history of Pen Computing (from a talk given to the Boston Computer Society in 1992) Dan Bricklin has posted a video of the presentation on youtube.com: Note that there is a tendency for major vendors to re-name Pen Computing ("Tablet PCs", "Touch PCs", "Organic User Interfaces" are a few such names), which tends to obscure the historical record a bit. Who invented Pen Computing? You may have read something somewhere that indicated some organization like Microsoft Nestor Apple Newton MicroPad IBM PenCept GO Corporation Rand Corporation Linus PenPoint CIC Wacom GRAIL Casio Slate Graffiti... had invented pen computing / tablet computing. After skimming through some of the references above, you might start to think some of those claims - particularly those by large corporations - could be slightly exaggerated: for 8:26:07 PM]

2 Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers example, inventing an improvement on spelling correction [Viterbi67] is not the same as inventing the first graphical word-processing program. Which is not to say that a particular system did not do some very interesting new things within the context of pen computing, such as [CarrR91a] and [PenCept85f]. Note: I find the very early references to technology interesting. For example, Handwriting recognition by a machine in real time, with a user writing with a stylus, goes back to before World War I: United States Patent 1,117,184, November 17, 1914 [GoldbergHE1914] United States Patent 1,311,384, July 29, 1919 [DrewWF1919] Optical character recognition (from a printed page) goes back to before the days of computers: United States Patent 1,838,398, December 29, 1931 [GoldbergE31a] Electronic Tablets were invented in the 19th century: United States Patent 461,472, October 20, 1891 [Gray1891b] United States Patent 491,347, February 7, 1893 [Gray1893b] (Note: Elisha Grey is best known in history as the person who may have invented the telephone before Alexander Graham Bell, but lost the patent dispute in a famous and controversial court decision) "Electronic ink" (the pen-computing kind, not the electronic display) is older than you think: "The RAND Tablet: A Man-Machine Graphical Communication Device", 1964 [DavisMR64b] (Note: This is the earliest published use of this term in pen-computing, AFAIK). Language knowledge being used in character recognition: The early references to the "Viterbi" algorithm are interesting to compare with more recent inventions on glyphs and context. [Viterbi67] So you think your recognition software is pretty good?... look at this. Another interesting slant is the many claims of "near perfect" handwriting recognition algorithms in the earliest days of pen computing, such as [Simek67] and [DimondTL57a] (Note: At one of my pen-computing jobs in the late 1980s, we joked that Dimond had achieved 97% "perfect" handwriting recognition in 1957, and that the technology had been getting about 5% better every decade since: the trend seems to have continued unbroken, like some odd form of Moore's Law.) You can find almost anything using one of the major search engines on the Internet these days. However, here are some links to other pages with talking about the history of pen computing. (Please note that links expire, so these may not all be up-to-date.) Images of early PenComputing products (a work in progress -- I have lots more in my attic). A talk on the "unknown" history of Pen Computing given to the Boston Computer Society in Schomaker's MIAMI group in the "Esprit" project includes a detailed taxonomy and another set of references relating to pen computing. Andreas Hennig has posted another (shorter) bibliography relating to pen computing. Nottingham Trent University has a research group in this area. Gary Perlman set up an extensive on-line bibliography on human-computer interaction. Bill Buxton has been working on haptic user-interfaces at various institutions for a long time. You can find his most recent work at The "Pen Report" has references to many of the commercial developments in one of the more recent pencomputing "crazes". It is listed as part of the AMUG Newton page, one of many user-group pages on the Web. There have been some original papers posted in odd places, such as this one by Scattolin Patrice. Of course, there is information on Wikipedia, but you should be cautious that the information can be biased towards a particular organization. Thank you, 8:26:07 PM]

3 Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Mirrors of this page can be found at: (Copyright :19:24 EDT) Back to homepage for Rueters-Ward / Zurück zur Heimatseite Rüters-Ward 8:26:07 PM]

4 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1878 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:17 EDT Mirrors of this page can be found at: (Generally most up-to-date) penhist (Prev page) Main Page (Next page) biblio75 [ARRL36a] American Radio Relay League, staff "The Radio Amateur's Handbook, 14th ed.", American Radio Relay League, 1936 Amateur radio electronics and theory [ARRL56a] American Radio Relay League, staff "The Radio Amateur's Handbook, 33th ed.", American Radio Relay League, 1956 (hardcopy book) Amateur radio electronics and theory [ARRL68a] American Radio Relay League, staff "The Radio Amateur's Handbook, 45th ed.", American Radio Relay League, 1968 (hardcopy book) Amateur radio electronics and theory [AbmaJS61a] Abma, John S.; Mason, Lawrence, J.; and Rice, David Reagan "Optophone", United States Patent 3,007,259, November 7, 1961 Reading devices for the blind: Optophones (auditory phones) known in the art. Probe with rollers on tip (for scanning text) and set of photocell sensors, sensors directly connected to auditory tone generator. [AlbergaCN67a] Alberga, C.N. "String similarity and misspellings", CACM, Vol 10 No 5, May 1967, pp Context string correction, spelling correction: review of several algorithms, some of which turned out to be trash [AndersonRH68a] Anderson, Robert H. "Syntax-Directed Recognition of Hand-Printed Two-Dimensional Mathematics", in "Interactive Systems for Experimental Applied Mathematics", pp , Klever, M. and Reinfelds, J., editors, Academic Press, New York, Also Proc. ACM Symposium on Interactive Systems for Experimental Applied Mathematics, 1967, pp Refers to min/max extents, "typographical" centroid of characters; User interface for mathematical symbol parsing, 2-D symbol parsing, sketch input [AndersonRH70a] Anderson, R.; Bator, R; Gagan, R.; Meads, J.; and Metrick, L. "Interactive Specification of Data Displays", NASA Contractor Report CR-1628, June 1970 Graphical display language: refers to a Symbol recognizer rather than a handwriting recognizer. Mentions Ledeen recognizer from a private communication at Harvard, Describes both fixed and 8:26:10 PM]

5 ( ) History of Pen and Gesture Computing adaptive/trainable recognizers (a bit naive about trainability). List set of 8 symbols (gestures?) for command input in one application, plus two others: e.g. Scrub/scratch-out for erase. [Applicon70a] Applicon Incorporated (Harry Lee) "Applicon CAD System with Trainable Hand-Drawn Symbol Recognition (video)", Applicon - Video available on YouTube, 2015 Marketing demonstration video of Applicon CAD/CAM system, ca Shows gestures on tablet: alpha gesture for zoom out, free-hand electronic ink lines straightened (sketch recognition). Macro definition with "teach" mode (records commands)., assign to any of fixed set of recognized gestures. [Applicon70b] Applicon Incorporated (Harry Lee) "Applicon Design Assistant System (video)", Applicon - Video available on YouTube, 2015 (three parts) Longer demonstration video of Applicon CAD/CAM schematic/architecture drawing system, ca Display is viewport onto larger virtual drafting table. Mentions use of gestures on tablet allow user to focus attention on display. Triangle gesture, inverted V (caret), Z gesture for zoom out, C gesture to center, R gesture (with second tap stroke) for resistor component, select gesture is diagonal line gesture on object. N, gestures. "Teach" mode assigns commands (command + X/Y position ) to gestures with location. Multilayer drawings for different classes of objects / views/ PC board layers. "Graphical entities" (objects), can be represented on multiple levels: editing definition changes all instances in drawing (compare with Smalltalk?). Snap-to grid. Macro-commands by single "stroke" (gesture) or single keyboard key. Text entered from keyboard, but can be edited with stylus gestures (no handwriting recognition). Shows drafting-table-size digitizer tablet (compare with tabletop?), electromechanical digitizer, for digitizing existing drawings by hand (movable keypad referred to as keyboard interface only). Dual displays: one for "world view" (Compare with Telesis) [ArkadevAG67a] Arkadev, A.G. and Braverman, E.M. "Computers and Pattern Recognition", translated from Russian by W. Turski and J.D. Cowan, Thompson Book Company, Washington, D.C., 1967 Papers on one character "invading the space" of another, compare to Write 1950 book on Arabic Numerals. Description of Perceptrons, early neural-net technology. [AttneaveF54a] Attneave, F. "Some informational aspects of visual perception", Psychology Review, Vol 61 No 3, 1954, pp Cited in Teh89, O'Callaghan70, cited in Lipscomb91 as Atteneave Features: Points of high curvature (corners) are dominant points in human pattern recognition of characters, not local maxima/minima for recognition [Baecker69] Baecker, Ronald M. "Picture-driven Animation", Proc. AFIPS Joint Computer Conference, May , SJCC, pp Sketch animation system: uses gestures? [BakisR68a] Bakis, Raimo; Herbst, Noel M.; and Nagy, George "An Experimental Study of Machine Recognition of Hand-printed Numerals", IEEE Trans. Systems Science and Cybernetics, Vol SSC-4 No 2, July 1968, pp Statistical adaptive classifier for handwriting: takes vectors of features, trains self to data base by finding best (lowest error confusion) weighting Data base collections: most are too small ("greenhouse data sets"), therefore low accuracy Formal training of user not acceptable for handwriting recognition 7000 and character collections from handwriting "back-room" forms not large enough Pre-processing of handwriting before recognition: scaling, shear normalization (tilt), Using too many feature measurements can lower performance, since the probability is higher that some features are of low effectiveness Gives actual/typical substitution errors from data collection on handwriting recognition [BeckerFK60a] Becker, Floyd K. "Electrographic Transmitter", United States Patent 2,925,467, February 16, 1960 resistive sheet telautograph/telewriting tablet with stylus, non-linear distortions from discrete terminals corrected by using multiple terminals along the edge, and generating a signal potential at each one that matches the linear voltage gradient. Storage tube display of both local writing, and of remote writing by another user, for collaborative work. Compare with later whiteboard systems? [BernsteinMI64] Bernstein, Morton I. "Computer recognition of on-line, hand-written characters", Memorandum RM-3753-ARPA, The RAND Corporation, Santa Monica, California October 1964 Also position and size invariant (no punctuation): Mentions that he does nothing on stroke segmentation Eight-direction chain direction codes (rotationally invariant), with endpoints for rotation-dependent added Features are eight-direction chain codes, quantized segment length, exhaustive table of types of sequences (like Penverter SME types?), followed by eight directions for end-point relative positions [BernsteinMI67a] Bernstein, M.I. "An On-Line System for Utilizing Hand-Printed Input Progress Report", System Development Corporation, Santa Monica, California, Report No NASA-CR-86108; TM/L/3052/001/00, 18 December 1967 Handwriting recognition software and user-interface: Appendix B refers to using spelling dictionary to do character segmentation / stroke parsing, and corner detection (and not just local minima/maxima?) [BernsteinMI67b] Bernstein, M.I. "Flow charts for recognizing hand printed characters on-line in real time operation", NASA Technical Report NASA-CR-86108; TM L , Dec 18, 1967 (abstract only) 8:26:10 PM]

6 ( ) History of Pen and Gesture Computing Handwriting recognition [BernsteinMI68a] Bernstein, M.I. and Howell, H.L "Hand-Printed Input for On-Line Systems: Final Report for Phase I", System Development Corporation, TM-(L)-3964/000/00, Santa Monica, California, April (abstract only) Trainable recognition: touch-display / electronic ink, by rear-projecting image onto digitizer tablet. Suggests free-form (2-Dimensional) input, not text entry is most useful application: implies that gestures are the useful application over handwriting recognition. [BernsteinMI68b] Bernstein, Morton I., and Howell, H.L. "Hand-Printed Input for On-Line Systems", System Development Corporation, TM-3937/000/00, Santa Monica, California, April 1968 (Abstract only). User interface: suggests handwriting best used for free-form/two-dimensional input, not linear text. [BernsteinMI68c] Bernstein, Morton I. and Williams, T.G. "A two-dimensional programming system", IFIP Congress, Edinburgh, Scotland, August 5-10, pp C84-C89 Cited in MillerGM69 on mathematical/flow-chart user interface parsing [BernsteinMI68d] Bernstein, Morton I. "A Method for Recognizing Handprinted Characters in Real Time", in Pattern Recognition, proceedings of IEEE Workshop on Pattern Recognition, L.N. Kanal, editor, Thompson Press, Washington, D.C., 1968, pp Smoothing, filtering, corner detection features, minima and maxima of character, types of component strokes (chain codes?) Handwriting tablet data rate: 2.5 msecs (400 points/sec.) [BernsteinMI69a] Bernstein, M.I., and Williams, T.G. "A Two-Dimensional Programming System", Proceedings 1968 IFIP Congress, North Holland Pub. Co, page 586, 1969 [BernsteinMI69b] Bernstein, M.I. and Howell, H.L "Hand-Printed Input for On-Line Systems: Final Report for Phase I", System Development Corporation, NASA Contractor Report CR-1284, March 1969 Trainable recognition: touch-display / electronic ink, by rear-projecting image onto digitizer tablet. Suggests free-form (2-Dimensional) input, not text entry is most useful application: implies that gestures are the useful application over handwriting recognition. Recognition uses 8-direction chain codes, zone recognition (and corner detection?) [BernsteinMI70a] Bernstein, M.I. "Software for Interactive Graphics Input", in Computer Handling of Graphical Information, Richard D. Murray, Editors, 1970, SPSE Press, pp Claims digitizing tablet invented by Dr. Herbert Teager of MIT around 1960, before Rand tablet Describes different tablet filter pre-processing algorithms: square-window, diamond-window, circular-window, doublesquare, etc. Handwriting recognition with electronic ink on a RAND tablet/display combination, [BitzerDL68a] Bitzer, D.L. and Skaperdas, D. "PLATO IV - An Economically Viable Large Scale Computer-Based Education System", National Electronics Conference, December 1968, pp Technical description of proposal for Version IV Plato graphical educational timesharing/teaching system. No reference to touchscreen (later?), graphics displayed from physical slides on 512x512 pixel transparent plasma storage display panel. [BlackwellFW67a] Blackwell, Frederick W. "An On-Line Symbol Manipulation System", Proceedings of the 22nd National ACM Conference, Thompson Book Company, Washington, D.C., 1967, pp Mathematical user interface for symbol-manipulation such as algebra: appears to use a keyboard for input, predecessor to Blackwell70 which was two-dimensional hand-written notation [BlackwellFW69a] Blackwell, Frederick W. and Anderson, Robert H. "An On-line Symbolic Mathematics System using Hand-Printed Two-Dimensional Notation", Proc. ACM '69 24th National Conf., pp Also RAND Memorandum RM-6018-PR, 1970 User-interface on handwriting and mathematics: parser, as well as recognition, can be extended by user (new operators, new syntax?) using Anderson68. Some user-interface questions on mathematics left open: how to show intermediate results; very large expressions. online manipulation of symbolic mathematical formulae: user hand-prints formulas, in ordinary two-dimensional notation. [BledsoeWW59a] Bledsoe, W.W. and Browning, J. "Pattern recognition and reading by machine", Proceedings of the East Joint Computer Conference, pp , December 1959 Had spelling dictionary with probabilities of words; compare to later context/dictionary solutions for handwriting recognition [BledsoeWW66a] Bledsoe, W.W. and Browning, J. "Pattern recognition and reading by machine", in Pattern Recognition, L. Uhr, editor, Wiley, New York, 1966, pp Cited in Srihari83 : same as Bledsoe59? [BlissJC70a] Bliss, J.C.; Katcher, M.H.; Rogers, C.H.; and Shepard, R.P. "Optical-to-Tactile Image Conversion for the Blind", IEEE Trans. on Man-Machine Systems, vol 11 no 1, March 1970, pp piezo-electric vibrators in finger scanner for the blind: suggests two sets of optics, one for letter-size page scanner, other for general environment. Compare with Opticon (accessibility) [Bonner67] Bonner, R.E. "Segmentation Method and Apparatus", United States Patent 3,344,399, assigned to International Business Machines Corporation, New York, New York, September 26, :26:10 PM]

7 ( ) History of Pen and Gesture Computing Segmentation/recognition of handwriting: Uses eight-vector-direction chain codes. Claims related to curvefollowing. [BoydLA1888a] Body, Lawson A. "Scriptoscope", United States Patent 389,190, September 11, 1888 Periscope/scriptoscope device, mirrors to allow typewriter operator to see typing happening on underside of platen. [Brenner70] Brenner, A.E. and debruyne, P. "A Sonic Pen: A Digital Stylus System", IEEE Trans. Computers, June 1970, pp Sonic digitizer using plane microphones: three-dimensional digitizer: states 1mm resolution. [BrownCD70] Brown, Curtis Del "A Real Time Handprinted Character Recognizer", M.S. Thesis, M.I.T, June, 1970 Mentions Graham as standard set of handwriting styles (for Morse code) intended to avoid conflicts in form handwriting recognition using properties of curvature and chain codes reviews of Teitelman "sequence" recognizer, Groner at Rand corporation chain-code recognizer, Graham at Sylvania [BrownRM64] Brown, R.M. "On-line Computer Recognition of Handprinted Characters", IEEE Trans. Electronic Computers, Vol EC-13 No 12, December 1964, pp alphanumeric and mathematic symbols, uses term "dynamic character recognition", stroke-direction recognition using zones. Cited in Litvin82, FoleyJD84, FoleyJD82. Used zone-based input device of conductive zones, compare with stylator (not a real tablet). [BushV45] Bush, Vannevar "As We May think", The Atlantic Monthly, July 1945 Vannevar Bush essay that introduced the concept of the Memex, a hyperlinked information archive. Other references show handwriting (recognition?) input, and a telautograph or digitizer tablet. May have been anticipated by Goldberg Patent in [Calabi69] Calabi, L. and Hartnett, W.E. "A family of codes for the correction of substitution and synchronization errors", IEEE Trans. Information Theory, Vol IT-15 No 1, pp Cited in Kruskal83 String correction, spelling correction distance [CaldwellSH59a] Caldwell, S.H. "The Sinotype - a machine for the composition of Chinese from a keyboard", Journal of the Franklin Institute, Vol 267 no 6, June 1959 Chinese Kanji input keyboard, using one keyboard stroke per character stroke. Notes that two types of strokes (vertical / horizontal) very high frequency (50%) in Chinese writing, with diagonal 75%: keys duplicated for either hand. No computer processor, all discrete hardware components, binary codes implemented electromechanically. Cited by H. Crane. [CameronSH67a] Cameron, S.H. et al "DIALOG: A Conversational Programming System with a Graphical Orientation", CACM, Vol 10 pp DIALOG - Illinois Inst Tech, Electrosketch Interactive math/geometry using graphics tablet: cited for setting up command/translation areas on a digitizer tablet in Eichen73. Dynamic/virtual keyboard, autocompletion of typed (handwritten?) commands. [CarlsonG66a] Carlson, Gary "Techniques for replacing characters that are garbled on input", Proc. SJCC, 1996, pp Dictionary connection of miss-recognized (or miss-typed) characters using digrams and trigrams. [CaseyR66a] Casey, R. and Nagy, G. "Recognition of printed Chinese characters", IEEE Trans. Electronic Computers, Vol EC-15, February 1966, pp Cited in Agui79 [ChampionML31a] Champion, M.L. and Gregg, J.R. "Gregg handwriting", Gregg publishing company, 1931; Bibliography by T.R. Davis 1931 reference to shorthand writing styles? [ChengG68] Cheng, G., Ledley, R.S.; Pollock, D.; and Rosenfield, A. "Pictorial pattern recognition", Thompson Book Company, Washington D.C., 1968 Cited in O'Callaghan70 [CheslerLG67a] Chesler, Leonard G. and Turn, Rein "The Application of On-Line Graphical Techniques for Programming and Operating a "Moving Network" Monitoring Display", RAND Corporation Memorandum RM PR, 1967 GUI description for RAND Graphic Input Tablet for creating a real-time moving network display of spacecraft checkout operations [ChodrowMM66] Chodrow, M.M., Bivona, W.A. and Walsh, G.M. "A Study of Hand-Printed Character Recognition Techniques", Rome Air Development Center Technical Report No RADC-TR , February 1966 Cited in MunsonJH68 as showing handwriting recognition in 1965 [ChristensenC67a] Christensen, Carlos "An Example of the Manipulation of Directed Graphs in the AMBIT/G Programming Language", Proc. ACM Symposium on Interactive Systems for Experimental Applied Mathematics, 1967, pp Visual programming. Compare with LabView, other AMBIT references for discussion of user interface on touchscreen. 8:26:10 PM]

8 ( ) History of Pen and Gesture Computing [ChuangPC70a] Chuang, P.C. "Recognition of Handprinted Numerals by Two-Stage Feature Extraction", IEEE Trans. on Systems Science and Cybernetics, april 1970, pp OCR static handwritten character recognition using two stages of feature extraction: low-level (line thinning, local coordinates of points, second state is branching and union (crossing?) points. [ClarkDW70a] Clark, D.W., Connell, David B., Dimeo, Michael P., Opitz, Bruck K., Sammon, John W. jr., and Sanders, Jon H. "Handprinted Character Recognition Techniques", NTIS Report AD , RADC Technical Report RADC-TR , October, 1970 (abstract only) On-line handprinted/handwritten character recognition, constrained to Standard Character Format. Claims 99% correct recognition (without rejection). [ClemensJK65a] Clemens, Jon Kaufman "Optical Character Recognition for Reading Machine Applications", Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, Massachusetts, 1965 Optical character recognition, generates artificial stroke sequence similar to on-line character recognition. Cited in MunsonJH68 as showing handwriting recognition in 1965 [ClowesMB68a] Clowes, M.B. "Transformational grammars and the organization of pictures", paper presented to the PISA Conference on Automatic Interpretation and Classification of Images, Pisa, Italy, August, 1968 Cited in O'Callaghan70 [ColemanML67a] Coleman, M.L. "Design for a text editor", Memo CCP 238, Carnegie-Mellon University, Pittsburgh, Pennsylvania, 1967 See also Coleman69 user-interface on gestures / handwriting symbols [ColemanML69a] Coleman, Michael L. "Text editing on a graphic display device using hand-drawn proofreader's symbols", from Pertinent Concepts in Computer Graphics: Proceedings of the 2nd University of Illinois Conference on Computer Graphics, Faiman, M. and Nievergelt, J., editors, University of Illinois Press, Urbana, Illinois, 1969, pp Also in M. Faiman and J. Nievergelt (Eds.), Pertinent Concepts in Computer Graphics User interface: gesture/command symbols for proofreading/editing. User-interface/gesture nine symbols for text: crossing out to erase, insert, transpose, move, remove spaces / close up, insert space, scroll up, scroll down, delete. Features for gestures / handwriting proofreading marks: closure. User-interface: text entry via voice, tolerate higher error rate on voice recognition if gesture-editor is convenient. [CollinsCC70a] Collins, Carter Compton "Tactile Television-Mechanical and Electrical Image Projection", IEEE Trans. on Man-Machine Systems, vol 11 no 1, March 1970, pp piezo-electric vibrators on back/torso scanner display for the blind: intent is viewing of general environment. Recognize faces as faces, but not distinguish different faces. 20x20 array (400 elements) [CostanzoRJ62a] Costanzo, Raphael J. "Touch controlled circuit", United States Patent 3,056,997, October 2, 1962 Capacitive touch sensor, using mutual capacitance: two interdigitated electrodes. [CraneHD60a] Crane, H.D. "Sequence Detection Using All-Magnetic Circuit", IRE Trans. on Electronic Computers, Vol EC-9, No 2 June 1960, pp Appears to be early version of SRI direction/accelerometer pen by Crane, with hard-wired circuits to respond to specific sequence of chain-codes. [CrookMN63a] Crook, M.N. and Kellogg, D.S. "Experimental Study of Human Factors for a Handwritten Numeral Reader", IBM Journal of Research and Development, January 1963, pp Cites G.G. Neal Wright. 90% correct recognition for somewhat constrained handwriting recognition on groups of subjects, no details on the type of recognition. Data scanned optically OCR. Subject were: students, sales clerks, sales clerks. Worst recognition success was with high-school males. [CurryJE69a] Curry, James E. "A Tablet Input Facility for an Interactive Graphics System", Proceedings IJCAI 1969, AFIPS/ACM International Conference on Artificial Intelligence, pp , May 1969 Trainable "symbol recognizer" for gestures for the TX-2 computer. Cited in RubinSM94 and Buxton05: Compare with Ledeen recognizer, gesture input? Example is a graphical editor for analog circuits (transistors, resistors, etc.) [DamerauFJ64a] Damerau, Fred J. "A technique for computer detection and correction of spelling errors", CACM, Vol 7 No 3, March 1964, pp Context via a spelling dictionary: cites to earlier work to 1960 [DannaSR69a] Danna, S.R. "Signature Identification Instrument", United States Patent 3,480,911, November 25, 1969, assigned to Conetta Manufacturing Co. Incorporated, Stamford, Connecticut. Signature verification using force/pressure [DavisMR64a] Davis, M.R. and Ellis, T.O. "The RAND Tablet: A Man-Machine Graphical Communication Device", AFIPS Fall Joint Computer Conference #26, part 1, 1964, Spartan Books, Baltimore, Maryland, pp Early reference to RAND tablet: Gray code electronics. See also historical notes in 2008 Ware paper. User interface/ ergonomics: says separate tablet and display not a problem for user after a couple of minutes, even for signatures: even has advantage that hand does not obscure drawing. See also interview notes with Uncapher User interface: earliest reference to electronic ink by name. Gestures? RAND Tablet digitizer was capacitive-coupled/electrostatic. 8:26:10 PM]

9 ( ) History of Pen and Gesture Computing [DavisMR64b] Davis, M.R. and Ellis, T.O. "The RAND Tablet: A Man-Machine Graphical Communication Device", Rand Memorandum RM-4122-ARPA, August 1964: ARPA Order No Early reference to RAND tablet [DayLF1910] Day, L.F. "Alphabets Old and New", B.T. Batsford, Limited, London, 1910 Examples of alphabet/character styles from historical sources: includes some examples of Greek, Coptic etc. alphabets, mostly ancient styles of Roman alphabet. [DeHaanWR66a] DeHann, William R. "The Bell Telephone Laboratories automatic graphic schematic drawing program", Proc. of DAC '66 SHARE design automation project Early CAD / schematic sketching program: input is by sketch recognition on coordinate paper, rather than touchscreen or lightpen digitizer [DeringerD68] Deringer, D. "The Alphabet", Funk and Wagnalls, New York, 1968 Background on historical variability of the written alphabet [Deutsch67] Deutsch, P. and Lampson, B.W. "An Online Editor", CACM, Vol 10 No 12, 1967, pp See also Coleman69 on gesture user-interface. QED keyboard-command text editor had two-letter/twocharacter macros/"gestures" for inserting stored phrases and text. [DevoeD67a] Devoe, D. "Alternatives to Handprinting in the Manual Entry of Data", IEEE Trans. Human Factors in Electronics, Vol HFE-8 No 1, January 1967, pp Cited in FoleyJD84. States that handprinting on forms (keypunch sheets) first step in data entry, regardless of final data entry (keyboard, mark sense forms, also compared with handwriting -- script?) [DevoeD68a] Devoe, Donald D., and Graham, Donald N. "Evaluation of Handprinted Character Recognition Techniques", Sylvania Electronics Systems Final Report No F ; RADC-TR , May 1968 (Abstract only). cited in Goodale83 User interface for handwriting: visual feedback has no effect on writing style, but is essential for editing and correcting Handwriting recognition: user perception of adaptive versus constrained is about the same (adaptive itself is constraining?) [DeweyG23a] Dewey, G. "Relative Frequency of English Speech Sounds", Harvard University Press, Cambridge Massachusetts, 1923 Cited in Srihari83: on spelling/context correction [DickensonWE60] Dickenson, W.E. "A Character-Recognition Study", IBM Journal of Research and Development, July 1960, pp Simulation (no real data) for recognition for ten "specially designed digits", using X-Y scans. Special digit shapes for OCR are intentionally rectangular. [DimondTL57a] Dimond, T.L. "Devices for reading handwritten characters", Proceedings of Eastern Joint Computer Conference, pp , December 1957 Early paper on handwriting character recognition: Stylator (stylus translator), a zone-based tablet, which was earlier than RAND digitizer tablet, which was full Cartesian coordinates. Describes cartesian-coordinate grid for character recognition, zone-based recognizers. [DimondTL58a] Dimond, Tom.L. "Experimental device for reading handwritten numbers", Electronic Equipment, Vol 6 No 1, January 1958, pp 6-7 Stylator zone-based handwriting recognition input device [DimondTL63a] Dimond, T.L. "Machine Reading of Handwritten Characters", United States Patent 3,108,254, October 22, 1963, assigned to Bell Telephone Laboratories, New York, New York. Early patent on five-zone/sector recognition. Constraints: user writes around two vertically-oriented dots with a stylus. See also Stylator. [DixonNF65] Dixon, N.F. "Subliminal Perception: The Nature of a Controversy", McGraw-Hill, 1965 (Date approximate): threshold for recognition in humans will vary [DoyleW60a] Doyle, W. "Recognition of sloppy, hand-printed characters", Proceedings of Western Joint Computer Conference, 1960, pp Original date 10 December 1959, Revised 2 May 1960 Cited in Genchi68 early OCR handwriting recognition for sloppy uppercase handprinting. Notes that adapting/training to real-world data can lead to different fit / evaluation than comparing from ideal representation. Feature extraction, then scoring. [DrewWF1919] Drew, W.F. "Electric Keyboard for Calculating Machines, Type Writing Machines, and the Like", United States Patent 1,311,384, July 29, 1919 on a keyboard terminal that uses a stylus to touch conductive key contacts: relate to digitizer technology, or handwriting input with a stylus. Cited in Irland64. [DreyfussH59a] Dreyfuss, Henry (Associates) "H1 Anthropometric data - Adult Male seated at Console", Henry Dreyfuss & Associates The Measure of Man and Woman: Human Factors in Design. Whitney Library of Design, 18 East 50th Street, New York 22, New York. Ergonomic figures for human reach, console/desk design [DreznerSM66a] Drezner, Stephon M., Gatto, O.T. and Wisneiwski, T.D. "Report on a Demonstration Test of a Computer-Assisted Countdown", RAND Corporation Research Report RM-5005-NASA, :26:10 PM]

10 ( ) History of Pen and Gesture Computing Describes user interface for a prototype stylus GUI, that enables the executive (user?) to act online with a simulation of the countdown: he can make changes to the script or to the simulation [DuMontAB35a] Du Mont, Allen B. "Telautograph", United States Patent 2,000,014, May 7, 1935 Telautograph using resistive-sheet digitizer (tablet) and CRT cathode-ray-tube display. Resistive sheet is carbon plate. Thirty-second persistence on storage scope display -- no explicit erase. [DudaRO68a] Duda, R.O. and Hart, P.E. "Experiments in the recognition of hand-printed text: part II -- context analysis", Proceedings of Fall Joint Computer Conference, 1968, Thompson Books, Washington, D.C., pp Context in handwriting (handprinted) recognition on Fortran coding sheets. Applying syntactic rules (GOTO 5 more likely than GOTO S), limited use of semantics (variable names unlikely to appear only once in a program), syntax of mathematical expressions. Cited in Tappert's bibliography, FoleyJD82. [DunnRankinP68a] Dunn-Rankin, Peter "The Similarity of Lower-Case Letters of the English Alphabet", Journal of Verbal Learning and Verbal Behavior, Vol 7, pp , 1968 Human recognition performance on lower-case letters. Analysis of critical/distinguishing features of letters: compare with Kuklinski and Shillman [DunnRankinP68b] Dunn-Rankin, Peter and Leton, Donald A. "Hierarchical Grouping Procedures for Comparing Indices of Letter Similarity", Perceptual and Motor Skills, Vol 27, 1968, pp Artificially/theoretically determine similarity (not a confusion matrix) of lower-case characters to determine which character pairs had similar feature. Contrasted with empirical results (with human subjects) matching which pairs characters were similar: similar results. [DunnRankinP68c] Dunn-Rankin, Peter; Leton, Donald A.; and Shelton Velma F. "Congruency Factors Related to Visual Confusion of English Letters", Perceptual and Motor Skills, Vol 26, 1968, pp Experimental study of which features of lowercase letters children learning to read are most reliable, which cause confusion. Break (O and C) cause little confusion, but rotation (e.g. p, b, d; n, u) is major cause of confusions. Simple comparison of similarity using transparent overlays good predictor of confusion matrix of pairs of characters. [DydykRB70] Dydyk, R.B., and Kalra, S.N. "Recognition of handprinting", Proceedings of 2nd International Conference on Systems Science, January 1969, pp Cited in Blatt88: 60% character recognition rate [EarnestLD62] Earnest, L.D. "Machine recognition of cursive writing", in "Information Processing", C. Cherry, editor, Butterworths, London England, 1962, pp Cited in MacDonald 1966, Tappert [EdenM61a] Eden, Mark and Halle, M. "The characterization of cursive writing", Information Theory, C. Cherry, editor, Butterworth, London, 1961, pp Cited in Sayre73 [EdenM62a] Eden, Mark "Handwriting and pattern recognition", IRE Trans. Information Theory, Vol IT-8 No 2, pp , 1962 Cited by Ehrich78 as saying: 33 basic stroke segments in cursive writing [EdenM64a] Eden, Mark and Mermelstein, P. "Experiments on computer recognition of connected handwritten words", Information and Control, Vol 7., pp , 1964 Yasuhara quotes his paper along with some Eden studies of cursive effects on handwriting styles: first author is Mermestein (see file) [EdenM68a] Eden, Mark "Other Pattern Recognition Problems and Some Generalizations", in "Recognizing Patterns", edited by Paul A. Kolers and Murray Eden, Massachusetts Institute of Technology Press, Cambridge, Massachusetts 1968, pp Any successful recognition machine will ultimately use principles derived from or relevant to the operation of living systems (implementation may differ) Circa 1968: Predictive value of most theoretical models for character recognition is near zero. Validation (testing) of theory has been very meager. (Note: Hype). General theory of pattern recognition based on human recognition and processes, not statistical models. Pragmatic pattern recognition (Murray Eden): depends onpurpose of classification. Why statistical methods for recognition will not hack it. Problems of sample bias and feature weighting in statistical/adaptive recognition. [EdisonTA1880a] Edison, T.A. "Electric-lamp", United States Patent 223,898, January 27, 1880 Original Thomas A. Edison electric light bulb. Carbon thread filament. [EdwardsAW64a] Edwards, A. Wood and Chambers, Robert L. "Can A Priori Probabilities Help in Character Recognition?", Journal of the ACM, Vol. 11, No. 4 (October, 1964), pp Using statistics on letter pairs as context, slight (but significant) improvement in OCR character recognition (using simple recognizer). Compare with Viterbi algorithm. [EinsteinA30] Einstein, Albert and Szilard, Leo "Refrigeration", United States Patent 1,781,541, November 11, :26:10 PM]

11 ( ) History of Pen and Gesture Computing U.S. Patent for German patent on a refrigerator by Albert Einstein and Leo Szilard: no moving parts. Not the original Swiss patent with Einstein's first wife Mileva. [EllisTO66a] Ellis, T.O. and Sibley, W.L. "On the Development of Equitable Graphic I/O", The RAND Corporation, Report P-3415, Santa Monica, California, July 1966 NTIS citation index: User interface with handwriting [EllisTO68a] Ellis, T.O., and Davis, M.R. "Digital computer and graphic input system", United States Patent 3,399,401, assigned to the United States of America, August 27, 1968 Cited in Tappert's patent list of 1986 Patent on putting a handwriting digitizer in front of a separate display in one system. Capacitive coupling digitizer tablet using wire grid Digitizing tablet with capacitively coupled stylus to X/Y grid, using Gray code encoding of position on on-line/dynamic handwriting recognition. Having touch surface separate from display screen advantageous: user adapted quickly, and display not occluded or obscured by hand in the way (integration of touchscreen and display) Cited in Fox88: on annotation drawing/handwriting [EllisTO68b] Ellis, T.O., and Sibley, W.L. "On the Problem of Directness in Computer Graphics", The RAND Corporation, Report P-3697, March 1968 Commentary to accompany a film (!!) entitled "The GRAIL Project". See also RAND history by Ware, 2008 System depends heavily on real-time symbol recognition: Functions can be invoked by pressing "virtual buttons" on the tablet. Bulk of paper is CRT display pictures. Precursor to work later named "direct manipulation"? Possible example of gesture commands? [EllisTO69a] Ellis, T.O., Heafner, J.F., and Sibley, W.L. "The GRAIL Project: An Experiment in Man-Machine Communications", The RAND Corporation, RM-5999-ARPA, Santa Monica, California, September 1969 Bernstein70 cited this on user-interface for flow-chart symbols User-interface: un-highlight/dim commands not valid at the moment Pen-computing GUI for creating and editing flowcharts Refers to a video of the handwriting recognition system in operation from 1968 Squiggle/rubout erase gesture. Flow-chart input on Rand Tablet, recognize rectangles, triangles, lines. Note: file folder also contains later references to the video of the Grail gesture/handwriting user interface, such as Alan Kay's talk. [EllisTO69b] Ellis, T.O., Heafner, J.F., and Sibley, W.L. "The Grail Language and Operations", The RAND Corporation, RM-6001-ARPA, Santa Monica, California, September 1969 Context sensitive (position) for manipulating on flowchart symbols vs. adding characters Handwriting user interface: multiple character erasure mark, caret gesture for insertion, writing on a "viewing window" Electronic ink / gesture user interface: has list of basic editing functions for handwriting (replacement, placement, insertion, deletion, erasure). Handwriting recognition for alphanumerics, flowchart parts, erase mark (zig-zag), special symbols. Virtual buttons in a handwriting/sketch input user interface, compare with Kaplow. Shows "scribble" (scratch out) gesture/character for erasure recognition, symbolic ink ("electronic ink") tracks Mathematical expression, flow-chart, and text editing (insert, move, delete, line detection, replacement) user interface with handwriting [EllisTO69c] Ellis, T.O., Heafner, J.F., and Sibley, W.L. "The Grail System Implementation", The RAND Corporation, RM-6002-ARPA, Santa Monica, California, September 1969 User interface for handwriting: display divided into different virtual areas Mathematical expression and text editing user interface with handwriting (early gestures?) Data structures to denote properties implied by the picture, plus positional information to relate stylus to the other forms Operating system description: garbage collection (or the avoidance thereof), real-time interrupt processing, etc. [EllisTO69d] Ellis, T.O. et al "Grail System", c2.com/cgi/wiki/wiki.cgi?grailsystem, fetched 2008 Wiki page (with links) on the Grail system and graphical programming languages: see other Ellis references 1969 [Engelbart70a] Engelbart, Douglas C. "X-Y Position Indicator for a Display System", United States Patent 3,541,541, November 17, 1970 Original (?) patent on a mouse, using two rotary wheels at right angles connected to rheostats [EnglishWK67a] English, W.K., Engelbart, D.C., and Berman, M.L. "Display Selection Techniques for Text Manipulation", Trans. of IEEE, Vol HFE-8, page 5-ff. Text selection UI with different devices: light pen (stylus digitizer), Grafacon (commercial version of Rand Tablet digitizer), joystick in rate and absolute modes, knee-control level, SRI "mouse". Based on two years actual experience. Mouse generally considered best. Cited in NewmanWM68a: as the original SRI "mouse", for a user-interface [EvansSutherland70a] Evans and Sutherland Computer Corp. "Evans and Sutherland Line Drawing System Model 1 System Reference Manual", Evans and Sutherland Computer Corporation, November 1, 1970, U Early graphical system: Windows and Views, 3D projections, drawing instructions similar to Turtle. Uses tablet input (LDS-1) [FBI64a] Federal Bureau of Investigation ""Louie Louie" (The 60's Song) Phonograph Record "Louie Louie" Distributed by Limax Music", U.S. Federal Be 8:26:10 PM]

12 ( ) History of Pen and Gesture Computing FOIA copy of FBI file (redacted) and lab reports on investigation of the lyrics of "Louie Louie". [FederJ65a] Feder, J. and Freeman, H. "Segment fitting of curves in pattern analysis using chain correlation", NTIS report AD619525, March, 1965 Cited in DavisLS77 on segment fitting for shape recognition: feature is match of chain codes along a segment [FingerFW47a] Finger, Frank W. and Spelt, David K. "The Illustration of the Horizontal-Vertical Illusion", Journal of Experimental Psychology, Vol 37, pp Cites earlier references for horizontal/vertical perception, back to 1912, [FittsPM54a] Fitts, Paul. M. "The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement", Journal of Experimental Psychology, Vol 47, pp , 1954 Fitts' Law. Original study was manual placement of pegs in holes, disks on pins. [Flannery1918] Flannery, J.R. and Dodds, E.I. "Means for Displaying Illuminated Words or Characters", United States Patent 1,285,098, assigned to Flannery Bolt Company, Pittsburgh, Pennsylvania, November 19, 1918 An electric pencil moved over contacts on a digitizer board to trigger display of characters with lamps [FluhrFR65a] Fluhr, F.R. "Light-Pencil Coordinate Positioner", NRL Memorandum Report 1592, March 1965, U.S. Naval Research Laboratory, Washington, D.C. Light-pen (?) device for use with radar display screens: functions by retrace insertion, generating marker at point of light-pencil on screen. [Frank69] Frank, A.I. "Data Entry System", United States Patent 3,487,371, assigned to ScanData Corporation, Norristown, Pennsylvania, December 30, 1969 Patent on real-time display of DCR input (?). Shows that final (throughput) error rate is 0%, if user can make corrections Real-time display makes DCR device self-teaching/adapting to train user immediate feedback [FreemanH61a] Freeman, H. "On the encoding of arbitrary geometric configurations", IRE Trans. Electronic Computers, Vol EC-10, 1961, pp Segment matching, for recognition of figures by segments [FreemanH70a] Freeman, H. "Boundary encoding and processing", in Picture Processing and Psychopictorics, B.S. Lipkin and A. Rosenfeld, editors, New York, Academic Press, 1970, pp Cited in Pavlidis80, Shu89 [FreibergerH63a] Freiberger, H. and Murphy, E.F. "Reading devices for the blind: an overview", Human Factors in Technology, McGraw-Hill, 1963, pp , earlier version in IRE Trans. HFE-2, March 1961, pp Cited in Rosenfeld as translating characters to sounds, but not character recognition. Optical probe: photoelectric cell an buzzer for blind to determine altitude of sun Reinvented multiple times by persons not aware of prior work. optophone, 1918 optical "recognizer" emitting tones for patterns of light and dark: compare with Opticon, Kurzweil reading machine. [FreibergerH66a] Freiberger, H. and Murphy, E.F. "Reading machines for the blind", Science, No 152, April 29, 1966, pp Cited in Rosenfeld as translating characters to sounds, but not character recognition. Reading machines/assistive technology for the blind. Early reading machine for digits, prototype used remote server over telephone connection. Mauch Laboratories reading machine: compare to later Kurzweil reading machine. [FriedmanSR68] Friedman, Symour R; Campbell, Douglas A; and Fehskens, Leonard G. "MOSAIC - The Improved Editing of Scientific Text by Handdrawn Commands and Data: A Technique for RAND Tablet and CRT Display", Technical Report ESD-TR Inforonics Inc, Cambridge MA, 14 Oct 1968 (abstract only) Nugent67 same title? Mentions cursive recognition, and use of both command input of a gesture and location of gesture, combining gesture and direct manipulation. Text editing of handwriting like van Raamsdonk? Gesture are select character (downward stroke), change word (wavey horizontal), change line (reverse pigtail), transpose (includes selection), insert (caret), capitalize / uncapitalize (letter C), carriage-return, space, backspace, tab (same as PenWindows gestures?). Two stroke (tap + stroke in direction) for punctuation comma/period,/etc. Compound gestures (double flick, etc.) [FrishkopfLS61a] Frishkopf, L.S. and Harmon, L.D. "Machine Reading of Cursive Script", Information Theory (symposium proceedings, London 1960). C. Cherry, Editor, Butterworth, London, 1961, pp Cited in Goodale83, Sayre73 [FrishkopfLS64a] Frishkopf, L.S. "Automatic Recognition of Handwriting", United States Patent 3,133,266, assigned to Bell Telephone Laboratories, New York, New York, 1964 Early script DCR recognition patent Script recognition on word basis, not letters, to get "context" X-ordinate is "general direction of writing" for script Cites disregard of position and size as a distinct advantage [FunkHL70a] Funk, Howard L. and Kamble, Stanley F. "Handwritten Character Recognition Apparatus", United States Patent 3,500,323, March 10, 1970, assigned to IBM Corporation, Armonk, New York. DCR with a mechanical pantograph tablet Six-zone/sector character recognition Cites real-time display for 8:26:10 PM]

13 ( ) History of Pen and Gesture Computing immediate feedback and correction Citeable in the Xerox/Palm patent dispute [Gaffney65] Gaffney, J.E. and Kusnick, A.A. "Character Identification Device", United States Patent 3,199,078, assigned to IBM Corporation, New York, New York. Like BLRT chain codes without regard to position (!) for handwriting recognition Compare to SRI pen with BLRT chain codes Number of strokes in different end-to-end slant orientations: used modified forms of letters O, X, and Y "point touch" to indicate end-of-character for character segmentation (!) [GainesHF39a] Gaines, Helen Fouche "Cryptanalysis: A Study of Ciphers and their Solution / Elementary Cryptanalysis", Dover Publications, 1939 / republished 1956 Classic text on cryptanalysis, decoding of encrypted data. Anagramming, etc. [GallensonL67a] Gallenson, L. "A graphic tablet display console for use under time-sharing", Proc. AFIPS '67 (Fall FJCC), November 14-16, 1967, pp Graphic tablet display (GTD) console user-interface with digitizer integrated into display. Higher data sampling rates (200 points/second) when stylus in contact with display for inking. Filtering of coordinate data, resampling for minimum distance between points. [GarnerWR54a] Garner, W.R. "Context Effects and the Validity of Loudness Scales", Journal of Experimental Psychology, Vol 48 No 3, 1954, pp Background on context showing median and mean (goodness and reaction time) are same for judgments [Genchi68] Genchi, Hiroshi, Mori, Ken-Ichi, Watanabe, Sadakazu, and Katsuragi, Sumio "Recognition of Handwritten Numerical Characters for Automatic Letter Sorting", Proceedings of the IEEE, Vol 56 No 8, August 1968, pp Handwritten numeral recognition for postal codes: features are zones after height and width normalization (after segmentation): 95% correct per character, 0.1% wrong across three digits (similar to human) Gives frequency of writing variants / base forms / prototypes with and without model characters shown. Linethinning, normalization of size: rejects touching characters Handwriting OCR: features are segment types, similar to chain codes, for scanned images [GilmoreJT58a] Gilmore, John T. and Peterson, Holmer P. "A functional description of the TX-0 computer", M.I.T. / Lincoln Laboratory Report 6M , October 3, TK7855.M41.L7455 The TX-0 computer was used for early stylus input, including handwriting recognition, before the GRAIL project [GlatteH59] Glatte, H. "Shorthand Systems of the World", The Wisdom Library, Division of Philosophical Library New York, 1959 (partial copy) Gabelsberger shorthand: unistroke alphabet [GoldB59a] Gold, B. "Machine recognition of hand-sent Morse code", IRE Trans. on Information Theory, Vol 5 No 1, March 1959 (abstract only) Recognizer for human-sent Morse code, 95% accuracy (defined as decipherable by a human reader) without the use of a word dictionary or other context heuristics. Compare with ARRL project mid 1960-s using diode memory? [GoldbergE31a] Goldberg, Emanuel "Statistical Machine", United States Patent 1,838,389, December 29, 1931 Very early OCR character recognition patent: Goldberg patent later mentioned as predecessor to Memex by Vannevar Bush, as it included filing and indexing of records. [GoldbergHE1914] Goldberg, Hyman Eli "Controller", United States Patent 1,117,184, November 17, 1914 Very early patent on handwriting recognition: conductive ink written on paper dots triggers adding certain weighted values for the dots. Happens to show constrained writing of numerals (around dot patterns), segmentation into what looks like chain codes. Cited in Tappert's patent list of 1986 on on-line/dynamic: handwriting recognition. See other patents by Goldberg [GoldbergHE1915] Goldberg, Hyman Eli "Controller", United States Patent 1,116,663, December 28, 1915 Very early patent on handwriting recognition: writing on 7-segment sensor for Roman letters and Arabic numerals (alphanumerics). See other patents by Goldberg [GolshanN70a] Golshan, N. and Hsu, C.C. "A Recognition Algorithm for Handprinted Arabic Numerals", IEEE Trans. Systems Science and Cybernetics, July 1970, pp Zone-based handwriting recognition: Features are like seven-segment LCD display for numerics: 1970 [GouldME36a] Gould, M.E. "Identifying Means", United States Patent 2,231,186, February 11, 1941 Zone/template-based optical object recognition: sensors/lamps in 5x10 grid are wired connected in particular patterns. When pattern of cells has input, circuit has enough power flowing to activate a relay. [Graham70x] Graham, D.N. "An Adaptive Character Recognition System", Sylvania Applied Research Laboratory, Research Note Number 718 Simplified writing style for Morse code training used as constrained writing: Reference to Morse Code writing style from American Radio Relay League? Cited in Brown 70 reference. [GrayE1888a] Gray, Elisha "Art of Telegraphy", United States Patent 386,814, July 31, 1888 Telautograph: very early precursor to electronic stylus and digitizing tablet. Elisha Gray also in patent 8:26:10 PM]

14 ( ) History of Pen and Gesture Computing dispute for priority with Alexander Graham Bell for telephone. [GrayE1888b] Gray, Elisha "Telautograph", United States Patent 386,815, July 31, 1888 Telautograph: very early precursor to electronic stylus and digitizing tablet. [GrayE1891a] Gray, Elisha "Telautograph", United States Patent 461,470, October 20, 1891 Telautograph: very early precursor to electronic stylus and digitizing tablet. [GrayE1891b] Gray, Elisha "Art Of and Apparatus for Telautographic Communication", United States Patent 461,472, October 20, 1891 Telautograph: very early precursor to electronic stylus and digitizing tablet. [GrayE1893a] Gray, Elisha "Telautograph", United States Patent 491,347, February 7, 1893 Telautograph: very early precursor to electronic stylus and digitizing tablet [GrayE1893b] Gray, Elisha "Telautograph", United States Patent 494,562, April 4, 1893 Telautograph: very early precursor to electronic stylus and digitizing tablet [GreaniasE57a] Greanias, E.; Hoppel, C.J.; Kloomok, M.; Osborne, J.S. "Design of Logic for Recognition of Printed Characters by Simulation", IBM Journal of Research and Development, Vol 1 No 1, 1957, pp 8 ff. OCR handwriting recognition of hand-printed digits and characters; simulated input using punch-card input of bitmaps of images; "proportional parts" using vertical (not horizontal) scanned input. [GreaniasE63a] Greanias, E., Meagher, P., Norman, R. and Essigner, P. "The recognition of handwritten numerals by contour analysis", IBM Journal of Research and Development, Vol 7 No 1, January 1963, pp Cited in O'Callaghan70, Groner66a [Griffin67] Griffin, Eugene H. and Gunn, James W. "End of Character Detector", United States Patent 3,305,832, assigned to Sperry Rand Corporation, New York, New York, February 21, 1967 Cited in Tappert88b: on character segmentation on cursive script segmentation (Does OCR character segmentation) OCR scanning character segmentation (?) [GronerGF66a] Groner, G.F. "Real-Time Recognition of Handprinted Text", Memorandum RM-5016-ARPA, The Rand Corporation, October 1966 Handwriting recognition on RAND tablet: curvature feature, corner detection, endpoints, height, width, center, aspect ratio, position relative to writing baseline (on screen) Segmentation in handwriting by X overlap, centers, spatial separation Performance: goes into details of 20 minutes user practice, information training: 82% to 92%, 81% to 93%, 90% to 96%, depending on user training/familiarity Shows actual confusion pairs: 7 and >, G and C, 5 and S, + and T, 0 and ) (???), 2 and Z, 1 and I, (-1CL, [-1CL, etc. User-interface for handwriting recognition/editing of FORTRAN coding forms Quick writing more distorted than usual training samples 53 symbols. Recognition tablet at 4msecs/point or 250 Hz. Multiple stroke characters were not order dependent, segmentation by relative location [GronerGF66b] Groner, G.F. "Real-Time Recognition of Hand-printed Text", AFIPS Fall Joint Computer Conference #29, 1966, Spartan Books, Washington, D.C., pp Mentions that subjective results of DCR are more important than statistics % correct [GronerGF67] Groner, G.F., Heafner, J.F., and Robinson, T.W. "On-line computer classification of handprinted Chinese characters as a translation aid", RAND Corporation publication P-3568, April IEEE Trans. Electronic Computers, Vol 16, December 1966/1967(?), pp Tappert's bibliography [GronerGF68a] Groner, G.F. "Real-Time Recognition of Handprinted Symbols", in Pattern Recognition, proceedings of IEEE Workshop on Pattern Recognition, L.N. Kanal, editor, Thompson Press, Washington, D.C., 1968, pp Description of GRAIL system (which see), user interface for handwriting recognition text input, erase gesture, special symbols [GronerGF68b] Groner, Gabriel F. "Real-Time Recognition of Handprinted Symbols: Program Documentation", RAND Corporation Research Memorandum RM-5550-ARPA, 1968 IBM/360 assembly language program for online graphical-input character recognition using the RAND tablet Recognizes handprinted letters, numbers, punctuation marks, and geometric figures: separates/segments characters written in rapid succession and close proximity [GroverGF66] Grover, G.F. "Real-Time Recognition of Hand-Printed Text", Proceedings of the Fall Joint Computer Conference, 1966, p 591 Cited in Bernstein70 on user-interface [Handel31] Handel, P.W. "Statistical Machine", United States Patent 1,915,993, June 27, 1933 OCR Optical Character Recognition system using rotating disk with stenciled patterns for characters [Hansell39] Hansell, C.W. "Multiplex Facsimile Printer System", United States Patent 2,143,875, assigned to Radio Corporation of America, Delaware, January 17, 1939 Facsimile system which read characters printed in one line along a roll of paper tape or other material. Second(?) earliest U.S. Patent on dynamic/on-line handwriting character recognition: refers to low error rate, both for wrong character and unrecognized / mis-recognized character. Cited in Irland64: on a 8:26:10 PM]

15 ( ) History of Pen and Gesture Computing handwriting terminal. [HarmonLD62a] Harmon, L.D. "Handwriting reader recognizes whole words", Electronics, Vol 35, August 1962, pp Tappert's bibliography on script handwriting recognition: script/dictionary handwriting recognition [HarmonLD63a] Harmon, L.D. "Method and Apparatus for Reading Cursive Script", United States Patent 3,111,646, November 19, 1963, assigned to Bell Telephone Laboratories, New York, New York. Refers to same subject/inter-subject variation in writing/segmentation Patent on script recognition segmentation Groups characters first by certain characteristics, such as vertical extent, then recognizes by other features within the group [HarmonLD64a] Harmon, L.D. "Automatic Reading of Cursive Script", United States Patent 3,127,588, March 31, 1964, assigned to Bell Telephone Laboratories, New York, New York. Word-at-a-time script recognition Script handwriting recognition in conjunction with a one-dimensional conductive-wire digitizing tablet Patent on script recognition segmentation Zone-based recognition of script using one-dimension horizontal digitizer Gives purely electro-mechanical implementation of handwriting/script input in the patent [HarmonLD64b] Harmon, L.D. "Automatic reading of printing and script", presented to the American Association for the Advancement of Science, 29 December, 1964 Cited in Sayre73 [HarmonLD64c] Harmon, L.D. "Automatic Character Recognizer", United States Patent 3,050,711, August 21, 1962, assigned to Bell Telephone Laboratories, New York, New York. Recognized characters by spiral scan, and noting the encounters with elements of the character: gives purely electro-mechanical implementation in the patent [HarmonLD65a] Harmon, L.D. and Sitar, E.J. "Method and apparatus for correcting errors in mutilated text", United States Patent 3,188,609, June, 1965 Cited in Suen79 Context correction in character recognition, using combined diagram probability with the two characters before and after character in question, including when a character is dropped/omitted (no recognition result) but place of a character is known. [HarperTP66a] Harper, T.P. "Electronic Handwriting Detection and Display Apparatus", United States Patent 3,271,515, September 6, 1996 Early light-gun/light-pen device. [Haverty68] Haverty, J.P. "GRAIL/GPSS: Graphic On-Line Modeling", RAND Corporation publication P-3838, Presented at an IBM Seminar on Operations Research in the Aerospace Industry: Models in Planning and Control, Newport Beach, April 1968 Describes GUI for logical flow diagram / flowchart input, to avoid delays of having input keypunched [HayesHV32] Hayes, H.V. "Radiant Energy Signalling", United States Patent 1,954,204, April 10, 1934 Very early patent on use of visible light wave transmission, which could be more directional (beam) than radio waves. Cited in Hieronymus patent on unknown "Psi-wave" radiation. [HeartFE70a] Heart, F.E.; Kahn, R.E.; Ornstien, S.M.; Crowther, W.R. and Walden, D.C. "The interface message processor for the ARPA computer network", AFIPS '70 (Spring) Proceedings of the May 5-7, 1970, spring joint computer conference SJCC, pp Arpanet IMP packet-switching router. Update routing tables every half second. Emphasis on support software: test harness for simulating host network traffic (on multiple IMPs), debugger built in on all operational units (not separate). [HelsonH65a] Helson, H. "Adaptation Level Theory, an Experimental and Systematic Approach to Behavior", Harper and Row, New York, 1964 User's perception changes to make data look "reasonable": use of context in human perception of images, such as handwriting recognition. [HieronymusTG49] Hieronymus, Thomas G. "Detection of Emanations from Materials and Measurement of the Volumes Thereof", United States Patent 2,482,773, September 27, 1949 Circuit for detecting unknown types of radiation (eloptic energy) using unknown physics to determine compositions of material from valence-electron characteristics at room temperature. Compare with Zimmerman 1995, Matias 2012, capacitive sensing of material. Cited in Analog Science article in 1960's by John W. Campbell. [HilditchCJ69a] Hilditch, C.J. "Linear skeletons from square cupboards", Machine Intelligence, Vol 4, 1969, pp Fast algorithm for thinning of lines, applications include trails in bubble chamber photographs, chromosome images, etc. [HladyAM69a] Hlady, A.M. "A touch sensitive X-Y position encoder for computer input", Proc. Fall Joint Computer Convergence, 1969, pp Surface acoustic wave transparent touchscreen, describes construction of transducers for radiating acoustic 8:26:10 PM]

16 ( ) History of Pen and Gesture Computing waves [HochbergJ53a] Hochberg, J. and McAlister, E. "A quantitative approach to figural 'goodness'", Journal of Experimental Psychology, Vol 46, 1953, pp Cited in IchikawaS84 [Hornbuckle67] Hornbuckle, G.D. "The Computer Graphics User / Machine Interface", IEEE Trans. Human Factors in Electronics, Vol HFE-8 No 1, March, 1967, pp Cited in FoleyJD84 [HoshinoT69a] Hoshino, T. and Kiji, K. "Computer-Aided Design for a Reader of Hand-Printed Characters", Proceedings IJCAI 1969, AFIPS/ACM International Conference on Artificial Intelligence, pp , May 1969 Dynamic/on-line handwriting recognition using a light pen for input, using bitmaps (?) of images and not directional stroke information. Uses confusion matrix to measure recognition performance, but states that confusion matrix gives no aid in finding the deficiencies in sequential logic (decision tree?) -- contrast with feature matrix and classification boundaries in Pencept/Shillman? Sequential logic designed by "trial and error", sequential logic evaluates features extracted separately: features appear to be chains of stroke segments. [Hough62] Hough, P.V.C. "Method and Means for Recognizing Complex Patterns", United States Patent 3,069,654, December 18, 1962 straight-line detection in images: division of television image into framelets such that straight-line segments can be extracted and recognized. Mentions recognition of could-chamber particle tracks. Secondary mention of handwriting analysis recognition and map (sketch) recognition. Cited in Li89 [HuMK62a] Hu, M.K. "Visual Pattern Recognition by Moment Invariants", IRE Trans. Information Theory, Vol IT-8, February 1962, pp Cited in Horowitz89 [Hughes68] Hughes, G.F. "On the mean accuracy of statistical pattern recognizers", IEEE Trans. Information Theory, Vol IT-14, 1968, pp Martin,GL90 on the accuracy of neural nets [HuntWA37] Hunt, W.A. and Volkmann, J. "The Anchoring of an Affective Scale", American Journal of Psychology, Vol 49, 1937, pp Recognition variability in humans affected by grouping of stimuli: As stimuli are varied as a group up or down a scale, human recognition boundaries shift to match [HuntWA38] Hunt, W.A. and Flannery, J. "Variability in the Affective Judgement", American Journal of Psychology, Vol 51, 1938, pp Variability in the affective judgment (vs generative variability). Recognition variability in humans increases with more stimuli (noise?) due to decrease in separation/differences among samples Functional attributes experiments: response time in humans decreases parallel to "goodness" judgement (decrease in affective judgement variability) [HuntWA41] Hunt, W.A. "Anchoring Effects in Judgement", American Journal of Psychology, Vol 54, 1941, pp Recognition variability in humans affected by grouping of stimuli. Psychophysics of human recognition variability [Irland64] Irland, E.A. and Morrison, C.G. "Real Time Reader for Hand-written Alpha Numeric Characters", United States Patent 3,142,039, assigned to Bell Telephone Laboratories, New York, New York, July 21, 1964 Extension of DimondTL63a patent for five-zone/sector recognition? to letters: 7-segment crossings / six zones. Says "D" normally written clockwise, "O" counter-clockwise Constrained handwriting to write around two dots one above the other: no punctuation or lower-case recognition, but full alphanumerics. [Ivaschentzo69] Ivaschentzo/Ivaschenko et al "Inductive Transducers for Graphical Input Devices", Soviet Journal of Instrumentation and Control, August 1969, pp Early tablet digitizer technology survey reference (in Russian) [JacksonJ1894a] Jackson, John "The Theory and Practice of Handwriting", William Beverley Harison press, 1894 Early work on handwriting styles (cursive). Mentions effects of writing desks -- compare with WardBJ thesis? [JohnsonCI68] Johnson, C.I. "Interactive Graphics in Data Processing: Principles of interactive systems", IBM Systems Journal, Vol 7 No 3/4, 1968, pp Mentions Sketchpad and light pens, primarily about graphics output software [JohnsonEA65a] Johnson, E.A. "Touch Display - a Novel Input/Output Device for Computers", Electronics Letters, Oct. 1965, Vol. 1 No. 0, pp Cross-wire electrostatic capacitive tablet, used in touchscreen. [JohnsonEA67a] Johnson, E.A. "Touch Displays: A Programmed Man-Machine Interface", Proc. Conf. on The Human Operator in Complex Systems. W. T. Singleton et al, Eds., London: Taylor and Francis, Ltd.: pp (partial copy) 8:26:10 PM]

17 ( ) History of Pen and Gesture Computing Expansion of 1965 Electronic Letters: de-bouncing of input, undo operation, pictures of touch architecture. [JohnsonEA69a] Johnson, Eric Arthur "Touch Displays", United States Patent 3,482,241, December 2, 1969 Cross-wire electrostatic capacitive tablet, used in touchscreen. Also mentions resistance. Works by mutual capacitance. Capacitance changes tuning of L/C circuit. [JohnsonRB56] Johnson, Reynold B. "Indicia-controlled Record Perforating Machine", United States Patent 2,741,312, April 10, 1956 Zone-based handwriting recognition terminal, using two guide dots, 10 digits. Illustrations show 80-column card input. Processing is electromechanical. Cited in Irland64: on a handwriting terminal [JohnsonTE63a] Johnson, Timothy E. "Sketchpad III: A Computer Program for Drawing in Three Dimensions", Proc. SJCC, 1963, pp D drawing using 2D touchscreen/light pen. User can rotate object to make any section "flat", draw drop line to another point to show where the Z position should be. Then user can draw any 3D line by drawing a 2D line with the light pen. (But what about drawing curved lines in 3D-space?) [JohnsonTE63b] Johnson, Timothy Edward "Sketchpad III: Three dimensional Graphical Communication with a Digital Computer", Master's Thesis, Mech. Eng. Dept., MIT, May D drawing using 2D touchscreen/light pen. Drawings are wire frame only: connected lines in 3-space. User can rotate object to make any section "flat", draw drop line to another point to show where the Z position should be. Then user can draw any 3D line by drawing a 2D line with the light pen. States there is a simple solution to drawing arbitrary surfaces, but not in this report. Low precision of light pen relative to drawing dealt with by cylindrical or spherical "snap-to" operation point on nearest line. [Kamentsky64] Kamentsky, L.A. "Character Recognition System", United States Patent 3,123,804, March 3, 1964 Zone-based constrained optical scanning OCR character recognition of handwritten digits on paper: user must write numerals around two vertically-aligned guide dots [KaplowR66a] Kaplow, R.; Brackett, J. and Strong, S. "Man-Machine Communication in on-line mathematical analysis", Proc. FJCC, 1966, pp Early graphical mathematical analysis program: compare with Macsysma/maxima. Graphical display, refers to briefly to graphical curve-tracing input, UI user interface mostly teletypewriter keyboard. [KatzD43a] Katz, David "Lecture and projection desk of the scriptoscope type", United States Patent 2,309,210, January 26, 1943 Overhead projector integrated with desk (compare with Weiller?), uses first-surface mirrors to avoid "parallax" distortion from front and rear surface of glass mirror. Scriptoscope also applied to telautograph/tablet? [KayAC68a] Kay, Alan C. "FLEX - A flexible extendable language", Master's Thesis, University of Utah, 1968 (published excerpt only) Alan Kay's FLEX predecessor to SmallTalk programming language, all program state always save -- no separate file system. [KayAC69a] Kay, Alan C. "The Reactive Engine", Ph.D. Thesis, University of Utah, 1969 (published excerpt only) Alan Kay's Dynabook thesis: published excerpt does not talk about touchscreen, but picture of "experimental display" shows IBM 1130 with tablet stylus or lightpen (hard to make out). [KimYT68a] Kim, Yung Taek and Evans, David C. "Syntax Directed On-line Recognition of Cursive Writing", U. Utah Comp. Sci. Technical Report 4-8, July 1968, ARPA Report AD Cursive handwriting recognition using Sylvania data tablet. Data filtered by minimum required distance between coordinate pairs (to reduce processing overhead), interpolation bit curve fitting for wild points. Stroke segmentation at local Y maxima (up/down stroke segments), recognition by syntactic tree. [KingJH66] King, J.H. jr. and Tunis, C.J. "Some Experiments in Spoken Word Recognition", IBM Journal of Research and Development, January 1966, pp Speech recognition for limited vocabulary of 30 words -- digits and mathematical operations -- using filterbank spectrum analysis. Claimed over 99% correct recognition for both speaker-independent (actually multi-speaker) and single-speaker system, but test data was the same as the training data. [Kinsler62] Kinsler, L.E. and Frey, A.R. "Fundamentals of Acoustics", 2nd Edition, page 503, John Wiley and Sons, Incorporated, New York, 1962 Speed of sound in air changes 0.08% for 5 degrees C (sonic/acoustic digitizer tablets) [KnollAL69a] Knoll, A.L. "Experiments with characteristic loci for recognition of hand-printed characters", IEEE Trans. Computers, Vol 18, pp , 1969 Choice of samples affects recognition performance results [KoenigW61a] Koenig, Walter Jr. "Telautograph System", United States Patent 3,005,050, October 17, 1961 Early resistive-sheet/film digitizer, with stylus: two sheets (one flexible, one conductive, one uniform resistance), analog signals only, not digital. Speaks of detecting coordinates of the stylus. Two resistive layers, one for each ordinate X/Y. [KostenL60a] Kosten, Leendert; van der Toorn, Adrianus Johannes; and Oberman, Roelof Maarten Marie "Information Bearer for Recording Figures in a Styled Form", United States Patent 2,963,220, December 6, :26:10 PM]

18 ( ) History of Pen and Gesture Computing seven-segment (stylized "8") printed pattern for constraining handwriting of numerals OCR handwriting recognition, alternative to marked-sense forms for scanning. [Krantz61] Krantz, D.L. and Campbell, D.T. "Separating Perceptual and Linguistic Effects in Context Shifts upon Absolute Judgement", Experimental Psychology, Vol 62 No 1, 1961, pp Human recognition variability: there are linguistic/semantic effects that change perception as well Human recognition variability: strongly affected by the most recent temporal stimuli [KuceraH67a] Kucera, H. and Francis, W.N. "Computational analysis of present day American English", Brown University Press, Providence, Rhode Island, 1967 Cited in Sinha88: as source of probability and statistics for context post-processing of English text [KuennapasT66a] Kuennapas, Teodor "Visual Perception of Capital Letters: Multidimensional ratio scaling and multidimensional similarity", Scand. Journal of Psychology, Vol 7, 1966 pp nine capital letters: I, J, L, F, E, U, O, S, A distinguished in human perceptual recognition by the characteristics of Rectangularity, O/Roundness, and I Vertical Linearity [KuepfmuellerK44a] Kuepfmueller, Kurt and Reche, Karl "Telautograph", United States Patent 2,338,949, January 11, 1944 Telautograph: electronic (CRT) display and resistive tablet, using a stylus, instead of electromechanical input and output or linear resistor arps. [KuhlF63] Kuhl, F. "Classification and Recognition of Hand-Printed Characters", IEEE International Convention Record, Part 4, 1963, pp Cited in Groner66a [LambrightJE70a] Lambright, John E.; Dow, Bruce R.; and Booker, Clyde A. Jr. "Interface Device and Display System", United States Patent 3,522,664, August 4, 1970 Resistive sheet touchscreen, rear-projection image. Diodes used to direct voltage across X or Y direction from AC signal. [LaningJH54a] Laning, J.H. Jr. and Zierler, N. "A Program For Translation of Mathematical Equations For Whirlwind I", Engineering Memorandum E-364, Instrumentation Laboratory, M.I.T., January 1954 Whirlwind computer: mathematical expression input and calculation using flexowriter (teletype-like) terminal. 2D mathematical input? Compare with later character-mode calculators with variables. [LeCaineH56a] LeCaine, Hugh "Touch Sensitive Organ", Fetched February 2011 Early touch-input device, organ keyboard detecting velocity of key being struck to affect tone played by electronic music organ, control of attack and volume. See also Young 1984 bio. [LeiferJC67a] Leifer, J.C., Mittelman, Louis Jr.; Sobol, Erwin J. "Free Stylus Position Locating System", United States Patent 3,342,935,September 19, 1967 Capacitive grid digitizer tablet with stylus. Fine and coarse position determination. States no tilt error because only tip of stylus is sensed. [Leitner61] Leitner, R.G. "Telescribing Apparatus", United States Patent 2,975,235, March 14, 1961 Resistive sheet/film digitizer using two sheets: a conductive flexible sheet and a second resistive sheet (or grid of conductors). Output device is mechanical: Telautograph [LesterCHT38] Lester, C.H.T. "Single-stroke writing", Bibliography by T.R. Davis 1938 reference to unistroke writing styles? [LevenshteinA66] Levenshtein, A. "Binary codes capable of correcting deletions, insertions and reversals", Soviet Physics - Dolkady, Vol 10 No 8, February 1966, pp Cited in Bozinovic82 spelling correction context (Russian): error correction for chains of binary codes (bits) that handle insertion errors, deletion errors, and substitution errors, not just substitution errors. [LewisHR68a] Lewis, Harry R. "SHAPESHIFTER: An Interactive Program for Experimenting with Complex-Plane Transformations", Proc ACM National Conference, pp Interactive mathematical modelling system: early CAD/CAM system two-dimensional mathematical expression recognition, handwriting recognition. Grafacon digitizer stylus used for menu picking, uses Ledeen recognizer with scrub/rubout/erase gesture, Anderson parse for hand-printed two-dimensional mathematics formulas. See Anderson Ph.D. thesis (cited). Cites to (informal) reference by Kenneth S. Ledeen [LickliderJCR62a] Licklider, J.C.R., and Clark, Weldon E. "On-Line Man-Computer Communications", Proceedings of the SJCC, May 1..3, 1962, pp Essay on power of interactive computing: military commanders, students, mathematicians. Sample systems are Tutor 1 (German/English vocabulary). Graph Equation, math exploration for students -- students needed typing skills, suggested longer term need for "some other symbolic input channel" (gestures? handwriting recognition?) Used light pen, not tablet. [LincolnA49] Lincoln, Abraham "Buoying Vessels over Shoals", United States Patent 6,469, May 22, 1849 Attaching additional floatation air chambers to a boat or ship to reduce the draft to get the vessel through 8:26:10 PM]

19 ( ) History of Pen and Gesture Computing shallow water. Only patent issued to a person who was also President of the United States. [LincolnLab65a] Lincoln Labs (M.I.T.) "Lincoln Lab TX-2 Circuit Sketch Demo (silent)", YouTube posting by Bill Buxton, Short video of Rand tablet and TX-2 computer for circuit sketching. Shows scrub-out erase gesture, X mark to select point in line to break and move, recognition of resistors. No text input. [LindgrenN65a] Lindgren, N. "Machine recognition of human language: Part III--cursive script recognition", IEEE Spectrum, May 1965, pp Cited in Sayre73 [LiuJ69] Liu, J. "Real-Time Chinese Handwriting Recognition Machine", E.E. Thesis, Massachusetts Institute of Technology, Cambridge Massachusetts, 1969 Cited in Goodale83 [LivermoreB63a] Livermore, Benjamin "Device for Hand Printing / Hand printing device", United States Patent 39,296, July 21, 1863 Early (19th century) chorded keyboard, fits into a pocket and can be operated by one hand (PDA?). Includes example of alphabetic character chording similar to 16-segment or 9-segment displays, or to unistroke/graffiti. [LongER60a] Long, E.R. and Henneman, R.H. "An Experimental Analysis of Set: Variables Influencing the Identification of Ambiguous, Visual Stimulus-Objects", American Journal of Psychology, Vol 73, 1960, pp Experiment on degraded/low-quality letter-patterns (dot-matrix characters): human recognition varies depending on what samples subjects were trained on Functional attributes experiments: tests of human character recognition using ambiguous stimuli. Compare with Shillman/Kuklinski. [LoomisHH60] Loomis, Herschel H. Jr. "Graphical Manipulation Techniques Using the Lincoln TX-2 Computer", Lincoln Laboratory, MIT, Cambridge Massachusetts Report , November 11, 1960 DavisMR64 mentions on handwriting user interface (?) User-interface (CAD) on TX-2 computer, using lightpen for sketching / drawing circuit diagrams: does not include character recognition, but suggests using character recognizer by H.P. Peterson (Page 8) : find a citeable a reference to that? [MIT52a] Digital Computer Laboratory, Massachusetts Institute of Technology "Project Whirlwind Training Program Material", M.I.T Collection of technical reports on Project Whirlwind: Memorandum M Short Guide to Coding; Report R-90-1 The Binary System of Numbers; Engineering Note E-479 Basic Conversion Program; Memorandum M-1734 WWI Control Switches and Pushbuttons; Memoranum M Programming for In-Out Units (Contains brief description of light-gun / early light-pen) -- supports multiple simultaneous light-guns (multi-touch?); Engineering Note E-516 Comprehensive System of Service Routines; Engineering Notes E-484 Policy on Outside Users of Whirlwind I. [MITRE63a] MITRE Corporation "Semi-Automatic Ground Environment (SAGE)", mitre.org, fetched 2010 SAGE system fully deployed in 1963: Early graphical input/output system using light-gun (invented by Bob Everett), precursor to light-pen and to computer mouse. [MacDonaldJS66] MacDonald, John S. "Experimental Studies of Handwriting Signals", M.I.T. Research Lab of Electronics, March 31, 1996, Technical Report RLE TR-443 Digitizer tablet consisted of electrolytic fluid and current sensors at each side of a rectangular tank. Contains model of handwriting motion implemented as a simulation. Asserts that modelling of neuromotor or physical generation of handwriting motion may assist in understanding invariant features for handwriting recognition: cites Teager and Mermelstein. [MarkeyHK42a] Markey, Hedy Kiesler and Antheil, George "Secret Communication System", United States Patent 2,292,387, August 11, 1942 early spread-spectrum encrypted communications (Inventor: Hedy Lamarr) for controlling torpedos and other devices. [MarrillWA63a] Marrill, T., Hartley, A.K., Evans, T.G., Bloom, B.H., Park, D.M., Hart, T.P., and Dorley, D.L. "CYCLOPS-1: A Second Generation Recognition System", AFIPS Conference Proceedings of the 1963 Fall Joint Computer Conference, Vol 24, pp 27-33, 1963 Light pen with handwriting recognition (single characters): Rhyne86 gives this as a reference to an early digitizer tablet, like RAND tablet [MartinWA65a] Martin, William A. "Sintax and Display of Mathematical Expressions", M.I.T. Artificial Intelligence Projet, Memo 85, Memorandum MAC-M-257, July 29, 1965 Automatic 2D layout of mathematical expressions (in Lisp). Lightpen (touchscreen) used to edit expressions, data structure has both positions/layout and parsing of expression, so that use can pick entire sub-expressions. [MartinWA67a] Martin, William Arthur "Symbolic Mathematical Laboratory", Ph.D. Thesis, Massachusetts Institute of Technology, Cambridge, Massachusetts, Report No MAC-TR-36, January 1967 Contains chapter on parsing two-dimensional mathematical expressions, perhaps with handwriting 8:26:10 PM]

20 ( ) History of Pen and Gesture Computing recognition input Points out that with handwriting user-interface, knowing when an input is complete is unclear (p 235) [MartinWA67b] Martin, W.A. "A Fast Parsing Scheme for Hand-Printed Mathematical Expressions", Massachusetts Institute of Technology, Artificial Intelligence Project Memo No 145, Memorandum MAC-M-360, October 19, 1967 User-interface for parsing mathematics: untested [MartinsonJL69a] Martinson, John L. "Method for Forming Written Symbols to be Read by Automatic Character Recognition Equipment", United States Patent 3,485,168, December 23, 1969 Zone-based recognition for handwritten digits: character must cross at corners (not segments) of sevensegment pattern. Recognized by simple optical sensors. [Masterson62] Masterson, J.L. and Hirsch, R.S. "Machine recognition of handwritten constrained Arabic numerals", IRE Trans. Human Factors and Electronics, Vol 3, pp 62-65, September 1962 Cited in Blatt88: 99.79% character recognition rate [McElwain62] McElwain, C.K. and Evans, M.E. "The Degarbler -- A Program for Correcting Machine-Read Morse Code", Information and Control, Vol 5 No 4, December 1962, pp Cited in Peterson80: spelling dictionary Early spelling corrector for context, tailored for typical missrecognition errors in machine-read Morse Code [MermelsteinP64a] Mermelstein, P. and Eden, M. "A System for Automatic Recognition of Handwritten Words", AFIPS Conference Proceedings of the 1964 Fall Joint Computer Conference, Spartan Books, Vol 26, pp , 1964 Cursive handwriting recognition, character segments at points of low velocity, classify segment as type of curve segment, recognize if can be generated from model for character writing styles. Shows photograph of electromechanical telautograph as type of tablet. Cited in Rhyne86 as early reference to a RAND-type tablet [MermelsteinP64b] Mermelstein, Paul and Eden, Murray "Experiments on computer recognition of connected handwritten words", Information and Control, Vol 7, 1964, pp Simulated (not actual) system for online handwriting recognition based on simple model of writing motion, segmentation, approximate matching, and some syntax/context constraints. No actual handwriting done? Cited in Sayre73 [MerrymanCT70a] Merryman, Coleman T. and Restle, Frank "Perceptual Displacement of a Test Mark Toward the Larger of Two Visual Objects", Journal of Experimental Psychology, 1970, Vol 84 No 2, pp Study of visual context affecting human perception/recognition of marks. [MikesG70] Mikes, George "The Land of the Rising Yen: Japan", Andre Deutsch, Publisher, Great Britain, 1970, Library of Congress Number Survey article of the quirks of Japanese Syllabic characters and language [MillerGA51a] Miller, G.A., Heise, G.A. and Lichten, W. "The intelligibility of speech as a function of the context of the test materials", Journal of Experimental Psychology, Vol 41, 1951, pp Cited in Long60: Human recognition/intelligibility of speech depends on context: greater intelligibility if limited set of test words, used in sentences, words repeated. [MillerGA56a] Miller, G.A. "The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information", The Psychological Review, Vol 63 No 2, March 1956, pp Wonderful early article on human perception: distinctive features, context, limits of absolute judgements, combined features, multiple dimensions of features [MillerGM69] Miller, G.M. "On-line recognition of hand-generated symbols", in Proceedings of Fall Joint Computer Conference, AFIPS, Vol 32, November 1969, pp Freeman code segments for handwriting recognition, 1969 Parsing strokes into segments, multiple-stroke characters: single-stroke characters a one-stroke character can also be a component of a multiple-stroke character Stroke segmentation using enclosing rectangle, overlap User interface for two-dimensional form [MinnemanMJ66] Minneman, Milton J. "Handwritten Character Recognition Employing Topology, Cross Correlation, and Decision Theory", IEE Trans. on Systems Science and Cybernetics, Vol SSC-2 No 2, December 1966, pp Handprinting recognition off-line, OCR, claims 98.5% recognition accuracy, test data was writing samples of single-stroke digits [MoodeyHC42a] Moodey, Hannah C. "Telautograph System", United States Patent 2,269,599, January 13, 1942 resistive-sheet/film digitizer for a Telautograph system, using AC voltages and sensor stylus [MoriKI70] Mori, Ken-Ichi; Genchi, Hiroshi; Watanabe, Sadakazu; and Katsuragi, Sumio "Microprogram Controlled Pattern Processing in a Handwritten Mail Reader-Sorter", Pattern Recognition, Pergamon Press, 1970, Vol 2 pp off-line handwriting recognition, digits only for postal/zip codes in Japan [MortonGA47] Morton, George A. and Flory, Leslie E. "Reading Aid for the Blind", United States Patent 2,420,716, May 20, :26:10 PM]

21 ( ) History of Pen and Gesture Computing Stylus pick-up unit with optical sensor, which translated degree of dark or light on the paper into an audible tone: sensor could have multiple elements [MunsonJH68a] Munson, J.H. "Experiments in the recognition of hand-printed text: Part I---Character recognition", Proceedings of Fall Joint Computer Conference, Vol 33, Thompson Books, Washington, D.C., December 1968, pp Samples of Fortran characters on coding sheets, with extra constraints. Forms data-entry [MunsonJH68b] Munson, John H. "The Recognition of Hand-printed Text", in Pattern Recognition, proceedings of IEEE Workshop on Pattern Recognition, L.N. Kanal, editor, Thompson Press, Washington, D.C., 1968, pp Cites handwriting recognition back to Handwriting recognition in context, FORTRAN coding sheets Features are concavities, contours, connected regions, spurs Says Clemens65 doesn't work [MyersTE60a] Myers, T.E. "Variable Resistance material", United States Patent 2,951,817, September 6, 1960 Conductive rubber: polyvinyl chloride flexible/pliable material with embedded metal conductive particles, becomes conductive under pressure [Nagy68a] Nagy, G. "Classification Algorithms in Pattern Recognition", IEEE Trans. Audio and Electroacoustics, Vol AU-16, No. 2, June 1968, pp (partial copy) Comparison of different statistical algorithms, tested on a training set of OCR characters [Nagy68b] Nagy, G. "State of the Art in Pattern Recognition", Proceedings of IEEE, Vol 56 No 5, May 1968, pp (hand-printed) data under actual working conditions likely to be much worse than test data [NagyG70a] Nagy, G. and Tuong, N. "Normalization Techniques for Handprinted Numerals", CACM, Vol 13 No 8, August 1970, pp Correction for optical distortion in OCR character scanning Rotational invariance loses distinction between 6 and 9 (critique of fully generalized pattern recognition?) [NassimbeneEG61a] Nassimbene, E.G. "Utensil for Writing and Simultaneously Recognizing the Written Symbols", United States Patent 3,182,291, August 25, 1961 DCR recognition with a photo/optical/light-sensing stylus on a tablet(!), position determined by amount of light falling on stylus sensor from illumination around the tablet: optical tablet [NeillJ69] Neill, J. "Numeric script mail sorter", Proceedings of automatic pattern recognition, pp 49-65, May 1969 Cited in Blatt88: 85%-99.7% character recognition rate [NeisserU60a] Neisser, Ulric and Weene, Paul "A Note on Human Recognition of Hand-Printed Characters", Information and Control, Vol 3, pp , 1960 Human recognition success rate is only 97% Study of human recognition testing using sign-in sheets (unconstrained characters) Gives human results for confusion-pairs in handwriting samples, "S-J" Has example of human difference between writer's intent and reader's perception for definition of correctness Strong recommended for any review of machine performance on handwritten characters [NewmanEB52a] Newman, E.B. and Gerstman, L.J. "A new method for analyzing printed English", Journal of Experimental Psychology, Vol 44, 1952, pp Statistical consistency of character in context in several language: some languages have redundant letters (can be removed without affecting amount of information content.) Cited in Suen79 on context statistics [NewmanEB60a] Newman, E.B. and Waugh, N.C. "The redundancy of texts in three languages", Information Control, Vol 3, 1960, pp Context statistics (letter pairs) have similar distribution in three languages with discrete but dissimilar alphabets: English, Russian, Samoan. Later alphabet generally correlates with greater redundancy. Cited in Suen79 on context statistics. Early work on statistical methods in language analysis. [NewmanWM68a] Newman, W.M. "A system for interactive graphical programming", 1968 Spring Joint Computer Conference, AFIP Conference Proceedings, Vol 32, Montvale, N.J. pp touch/drag (and select/tap?) interactions with a lightpen stylus Cited in Wallace76 for "virtual devices" [NewmanWM68b] Newman, W.M. "A Graphical Technique for Numerical Input", Computer Journal, Vol 11 No 1, May 1968, pp Cited in FoleyJD82 Stylus/pen user-interface: "Light Handle" user-interface recognizes clockwise/counterclockwise spirals as "shaft turning" spiral/circular gestures to control numeric value. Compare with later pie menus/marking menus? [NishidaN70a] Nishida, Nobou and Sakaguchi, Mitsuhito "Hologram Graphic Data Tablet", United States Patent 3,658,402, April 25, 1972 Optical tablet with small hologram cells on a plate, each hologram encodes its position, position determined by light coded value in light. Requires mechanical alignment of light source to transmissive plate. [NugentWR66a] Nugent, W.R. and Buckland, L.F. "Improved Text Editing using Hand-Drawn Commands and Data: A Technique for RAND tablet and CRT Display", Inforonics, Inc., First and Second Quarterly Progress Reports, Cambridge, Massachusetts, August-November 1966 Cited in Bernstein70 on user interface for text editing with handwriting recognition Friedman 68 same title? 8:26:10 PM]

22 ( ) History of Pen and Gesture Computing Mentions cursive recognition, and use of both command input of a gesture and location of gesture, combining gesture and direct manipulation? Text editing of handwriting like van Raamsdonk? [NugentWR67a] Nugent, W.R. and Buckland, L.F. "MOSAIC - The Improved Editing of Scientific Text by Handdrawn Commands and Data: A Technique for RAND Tablet and CRT Display", Third Quarterly Report, Inforonics, Cambridge, Massachusetts, 1967 Same reference as FriedmanSR68? Cited in Coleman69 on gesture / handwriting user interface, text input and editing using gestures (?) in [OCallaghanJ68] O'Callaghan, J. "Pattern recognition using some principles of the organism - environment interaction", Ph.D. Thesis, A.N.U., Canberra, Australia, September 1968 O'Callaghan70: early handwriting recognizer [OCallaghanJ70] O'Callaghan, J.F. "Problems in on-line character recognition", in "Picture Language Machines", S. Kaneff, editor, Academic Press, New York, 1970 Broad review of thinking on handwriting recognition, 1970, discussion from a conference Contains long Q/A on O'Callaghan's handwriting recognition: segmentation, features, performance time, rotation/tilt: feature space (for binary comparison) is a broad hump, not a fine line. Mentions problem of mal-adaptation to bad data in trainable adaptive recognizer [OliverBM64a] Oliver, B.M. "Time Domain Reflectometry", Hewlett-Packard Journal, Vol 15 No 6, February 1964, pp 1..8 Time domain reflectometry: Determine position of discontinuity in a cable/conductor by measuring echo time for gigahertz pulse. Technique later used in touch sensors. [Orr68] Orr, N.W. and Hopkin, V.D. "The role of the touch display in air traffic control", The Controller, Vol 7, 1968, pp 7-9 Cited in Beringer89 on electronic ink, integrated tablet/display [PattonR69] Patton, Robert "Graphic Data Tablets", The Electronic Engineer, November 1969, pp Cites Dr. Michael Pilla, Bell Labs Human Factors, on writing on separate tablet and display as not a problem. [PeckerEA63a] Pecker, Edwin A. Zone-based recognizer for handwritten digits using a single "+" arrangement of four conductive lines. [Peters63] Peters, R.D. "Signature Identification Device", United States Patent 3,113,461, December 10, 1963 Signature verification device using electro-mechanical meters to measure writing time, X and Y position (velocity?) via meters, which might be some other kind of display. Writing pad appears to be a resistive sheet digitizer with a tethered stylus. See also Herbst76 on signature verification using force vs. time. [PostmanL45] Postman, Leo and Miller, G.A "Anchoring of Temporal Judgments", American Journal of Psychology, Vol 58, 1945, pp Absolute scale and absolute judgment doesn't exist: depends on context of human recognition / perception Cited on Kuklinski's Ph.D. thesis, how an anchor affects perception of graphical context [PoultonEC65a] Poulton, E.C. "Letter differentiation and rate of comprehension in reading", Journal of Applied Psychology, Vol 49, 1965, pp Cited in Suen86 as study of inherent (to humans) legibility of different writing styles. Compare with Pencept pair-wise comparison [PrattF42a] Pratt, F. "Secret and Urgent", Blue Ribbon Books, Garden City, New York, 1942 Cited in Fisher75: frequency of letters in English text [Priver67] Priver, Arthur S. and Boehm, Barry W. "Curve Fitting and Editing Via Interactive Graphics", RAND Corporation Research Memorandum P-3742, Present at the ACM Symposium on Interactive Systems for Experimental Applied Mathematics, Washington, August 1967 RAND Tablet stylus to enter a curve, and try various ways of fitting, editing and displaying the curve, with Groner's character recognition scheme for alphanumeric inputs [RAND61] RAND Corporation "50 Years of Looking Forward", RAND Review, Fall Available at In 1961, researchers create the RAND Tablet, the first two-dimensional writing surface that allows humans to communicate instantly with computer through characters printed on a tablet. (Note: Stylator may actually have been earlier). JOSS interactive computer system [RankinBK65a] Rankin, B. Kirk III, Sillars, Walter A., and Hsu, Robert W. "On the Pictorial Structure of Chinese Characters", National Bureau of Standards Technical Note 254, January 4, 1965 Syntactic rules for Chinese characters: composed radicals and nodes. Study of what frequent combinations of strokes should be recognized as linguistic units. Grammar implementation similar to Chomsky' phrasestructure grammars (generative model for Chinese characters). [RankinBK67a] Rankin, B. Kirk III, Siegel, Stephanie, McClelland, Ann, and Tan, James L. "A Grammar for Component Combination in Chinese Characters", National Bureau of Standards Technical Note 286, 1967 Syntactic rules for Chinese characters: composed radicals and nodes. Study of what frequent combinations 8:26:10 PM]

23 ( ) History of Pen and Gesture Computing of strokes should be recognized as linguistic units. Grammar implementation similar to Chomsky' phrasestructure grammars (generative model for Chinese characters). [RankinK70a] Rankin, Kirk and Tan, James L. "Component Combination and Frame-Embedding in Chinese Character Grammars", National Bureau of Standards Technical Note 492, February 1970 Syntactic rules for Chinese characters: composed radicals and nodes. [RenggerRE68a] Rengger, R.E. and Parks, J.R. "A Survey of Handprinting", National Physical Laboratory, Ministry of Technology, Autonomics Division, March 1968 Measure of skew angle from vertical of handprinted/handwritten characters Context: written "I" in middle of sentence usually done without serifs, but if alone, seriffed. Handwriting/handprinting: variability caused by incomplete closure, extra curves and crossovers, misaligned limbs Contains handwriting samples/data collection: 55 writers (hands) not enough to get all likely variability [RhoadsKG70a] Rhoads, Kevin G. "SYSHAK: or, what a crock OS/360 is", SIPB (Student Information Processing Board), Massachusetts Institute of Technology, 1970 Class notes on exploits of security weaknesses in IBM OS/360 Operating system, mostly to put program into supervisor mode. [RippyDE65a] Rippy, D.E. and Humphries, D.E. "MAGIC: A Machine for Automatic Graphics Interface to a Computer", Proc. FJCC 1965, pp Early graphical display subsystem using special-purpose computer and voice-grade communications lines: light pen used to generate locator (cursor) on display, when locator (cursor) at correct position, separate manual switch to select. Refers to locator "servoing" to the light pen using up/down counters. Interruptbased subroutines for interactions. [RobbinsMF70a] Robbins, M.F. and Beyer, J.D. "An Interactive Computer System Using Graphical Flowchart Input", CACM, Vol 13 No 2, February 1970, pp User interface for flowchart definition using handwriting: no symbol recognition. Focus is on simulation of circuit of the flowchart (?) diagram and showing simulated output on an oscilloscope. [Roberts62] Roberts, Lawrence G. "Recent Development in Optical Character Recognition at M.I.T.", in "Optical Character Recognition", Spartan Books, 1962 Mentions early Teager electromagnetic grid tablet ("electronic drafting board"), with simple singlecharacter recognition Eden and Halle, only 18 strokes / stroke-types used to construct/write all English characters (??), simulation of persons handwriting by parameter variation: compare with later work on graphonomics Early dictionary-based recognition of handwriting using dictionary of strokes [RobertsLG66a] Roberts, L.G. "The Lincoln Wand", Fall Joint Computer Conference 1966, Spartan Books, Washington D.C., pp Ultrasonic/acoustic digitizer, digitizes in three 3-D dimensions, used as pointing device [RomeroA62a] Romero, Adolph "Electric Writing Board", United States Patent 3,019,425, January 30, 1962 Essentially a DC multi-touch/electronic ink switch tablet. Electric writing board supporting erasable electroconductive writing. X/Y grid of conductive points, each point matched to a lightbulb in an electric sign (both have the same grid). Writing stylus/pencil/pen leaves electroconductive material on the ends of the electrodes where they are exposed on the grid, so the writing shows up on the electric sign as the corresponding bulbs turn on. Erase by removing the electroconductive material. Can also write with conductive ink/material on paper, and put the paper face-down on the exposed electrodes. [RomeroA64a] Romero, Adolph "Stylus Controlled Sign with Contact Grid of Parallel Plates", United States Patent 3,128,458, April 7, 1964 Better contact grid for electric writing board of RomeroA62a [RosenfeldA69a] Rosenfeld, Azriel "Picture Processing by Computer", Computing Surveys, Vol 1 No 3, September 1969, pp Survey of early techniques for computer processing of pictures by translating to a picture description language / picture language. Image compression includes describing fixed uniform regions, or short-code encoding of most frequent grayscale pixel values, not just approximations. Refers to handwriting recognition, pointing out that it is not strictly OCR, uses dynamic information. [RosenthalAH47a] Rosenthal, Adolph H. "Two-way Television Communication Unit", United States Patent 2,420,198, May 6, 1947 Very early videoconferencing system, using half-silvered mirror and television/video camera to let use look directly at screen for eye contact. Compare to ClearBoard. [RovnerPD69a] Rover, P.D. and Henderson, D.A. Jr. "On the implementation of AMBIT/G: a Graphical Programming Language", Proc. IJCAI 1969: 9-20 Graphical editor for 2D two-dimensional visual programming language AMBIT/G. Recognized handwritten Symbols used to modify program drawing or as system commands (gestures), also light targets (menu/icon). Long lists of light targets are scrolled/turnpage to next portion. Cited by Buxton, 1969 gesture recognition, for using the TX-2 "recognizer". 8:26:10 PM]

24 ( ) History of Pen and Gesture Computing [RuddiesGH66] Ruddies, Guenther H. "Deine Handschrift / Dein Ruin: Mach mehr aus Deiner Schrift", Verlag und Druckhaus Dr. Jenner, Munich (hardcopy book) "für Deine graphologische Beurteilung bei Bewerbung, beruflichem Aufstieg und Partnerwahl" Describes use of graphological evaluation for employment: describes individual handwriting characteristics: compare to some aspects of graphonomics by Teulings? [SayreKM65a] Sayre, Kenneth M. "Recognition: A Study in the Philosophy of Artificial Intelligence", Notre Dame, Indiana, University of Notre Dame Press, 1965 Cited in Goodale83 [ScriptureEW1900a] Scripture, Edward W. "Studies from the Yale Psychological Library", Yale University, New Haven, Conn Contains "Researches on movements used in writing" by Cloyd N. McAllister: Has an electronic (electric) writing tablet, regularly generated sparks from a writing stylus left equally-timed dots on the paper while subject was writing. Cites French study that connected writing is *slower* than hand printing -- because of stylish writing? Notes on baseline of writing. [SearsFW60a] Sears, Francis Weston and Zemansky, Mark W. "College Physics: Mechanics, Head and Sound. Third Edition", Addison-Wesley, 1960 (hardcopy book) Standard physics reference and textbook [SerrellR68a] Serrell, Robert and Kling, Frederick R. "Response expression apparatus for teaching machines", United States Patent 3,382,588, May 14, 1968 large-format (wall-sized) touchscreen with rear projection of images from film, intended as teaching machine for children. individual (mutual) capacitive touch sensors, each individually wired for discrete touch points on display. Transparent conductive plates (electrodes) of ITO, ATO, or conductive glass conductors. [ShannonCE49a] Shannon, C. E. "Communication Theor of Secrecy Systems", Classified Confidential Report "A Methematical Theory of Cryptography", declassified 1949 A priori and a posteriori probabilities of content sequences with symmetrical encryption/decryption keys, as a measure of the quality of a cypher key. [ShannonCE51a] Shannon, C. E. "Prediction and entropy of printed English", Bell System Technical Journal, Vol 30, 1951, pp Early study on dictionary of frequent pairs in English, applied to compressive encoding of text based on encoding next character based on statistical likeliness. Cited in Suen79 on dictionary statistics. [ShawAC70a] Shaw, Alan C. "Parsing of Graph-Representable Pictures", Journal of the A.C.M., Vol 17 No 3, July 1970, pp Parsing of pictures (actually black/white line diagrams: block letters, electrical circuit, etc.) into primitive components (small drawing elements) and grammar for specifying how the elements are connected. One example of a photograph, but image reduced to black/white and illegible. Cited in FoleyJD82 [ShawJC58a] Shaw, J.C., Newell, A., Simon, H.A., and Ellis, T.O. "A Command Structure for Complex Information Processing", RAND Corporation, Santa Monica, California, Report No P-1277; PB Also cited as [JCC 13] Proceedings of the Western Joint Computer Conference, May, 1958 Mathematical expression parsing user-interface? [ShepardDH53] Shepard, D. H. "Apparatus for Reading", United States Patent 2,663,758, December 22, 1953 OCR Optical Character Recognition system using mechanical flying-spot scanner with rotary drum [SheridanTB63a] Sheridan, T.B. and Ferrell, W.R. "Remote Manipulative Control with Transmission Delay", IEEE Trans. on Human Factors in Electronics, Sep. 1963, Vol 4 No 1, pp (partial copy) User-interface responsiveness: if less than 45 milliseconds between event and system response, perceived as no delay. If greater than 300 milliseconds, perceived by user as sluggish, resulting in move-and-wait behavior by user. [Simek67] Simek, J.G. and Tunis, C.J. "Hand-printing input device for computer systems", IEEE Spectrum, Vol 4, 1967, pp Claims handwriting is faster than typing (??) Reports over 98% accuracy with a specially-trained subject (the author) [Sitar61] Sitar, E.J. "Machine recognition of cursive script: the use of context for error detection and correction", Bell Telephone Laboratories, Murray Hill, New Jersey, Memorandum for file, 12 September 1961 Cited in Sayre73 [SmithLB70] Smith, Lyle B. "A Survey of Interactive Graphical Systems for Mathematics", Computer Surveys, Vol 2 No 4, December 1970, pp Very early graph-fitting algorithms, etc. Mentions light pen as input device to some mathematical system. References to early tablets: RAND tablet, "graphic tablet display" (tablet and display integrated) from System Development Corporation 1967, "beam pen" input/output device. One reference to handwriting recognition of characters (Mermelstein 1964) Cited in FoleyJD82 8:26:10 PM]

25 ( ) History of Pen and Gesture Computing [SmithlineET64a] Smithline "Limited Vocabulary Script Reader", IBM Technical Disclosure Bulletin, Vol 7 No 6, November 1964, pp Cited by Crane75 (partial copy) [SpinradRJ65a] Spinrad, R.J. "Machine recognition of hand-printing", Information and Control, Vol 8 No 2, 1965, pp Hand-printing recognition rate 93.5% -- not sure how measured. Error charts appear similar to Shillman confusion matrix. Refers to reconstruction of mutilated text in English: spell checker / spelling correction? [StevensME61a] Stevens, M.E. "Automatic Character Recognition - A State-of-the-Art Report", U.S. Department of Commerce, Washington, D.C., NBS Technical Note 112, May 1961 hand-written numerals recognition using vector crossing technique : a standardized single-stroke numeric character style tables of frequency of occurrence of upper-case and lower-case characters Very extensive bibliography of handwriting recognition as of Cited in Groner66a [StevensME70a] Stevens, Mary Elizabeth "Information Acquisition, Sensing, and Input: A Selective Literature Review", U.S. Department of Commerce, Washington, D.C., NBS Monograph 113-1, March 1970 Extensive review of handwriting, voice, and optical character recognition as of Cites numerous references: Ledeen and Wolfburg recognizer, Simek, Data Trend "MIMO" (Man-in/Machine-out) handwriting recognition platen (digitizer) as of 1965, Burger 1964 sketch/gesture recognition for chemical symbols, Goldberg 1931 patent, etc. Several references to automatic handwriting recognition and interactive computer user interfaces on tablets in early 1960s. [StevensSS58a] Stevens, S.S. "Adaptation-level vs. the Relativity of Judgment", American Journal of Psychology, Vol LXXI No 4, December 1958 Classic study on context and adaptation affecting human perception/recognition [StoneL54] Stone, Leo "On the Principal Obscene Word of the English Language -- An Inquiry, with Hypothesis, Regarding its Origin and Persistence", International Journal of Psycho-analysis, 1954, Vol 35, pp (abstract only) Scottish in origin. See also Fairman [StoneRC1889a] Stone, Ross C. "Telegraphic Apparatus", United States Patent 400,141, March 26, 1889 Precursor of electric tablet: conductive wired stylus can be dragged across holes (dashes and dots) in perforated plate at a uniform rate of speed to send Morse code, Morse characters arranged in appropriate keyboard or other configuration for easy use. [StotzR63a] Stotz, R. "Man-Machine Console Facilities for Computer-Aided Design", Proceedings of AFIPS 1963 Spring Joint Computer Conference. Cited in DavisMR64 on early handwriting user-interface? Not clear about recognition or gestures. Early hardware for CAD design: description of light pen, display storage scopes graphic display. Light buttons -- on screen menus and actions, can be embedded in picture being displayed: compare icon. [SutherlandIE63a] Sutherland, Ivan E. "SKETCHPAD: A Man-Machine Graphical Communication System", Ph. D. thesis, Massachusetts Institute of Technology, January 1963 Shows early light-pen digitizer design; refers to "pseudo-pen location" for editing graphics, locate nearest object. [SutherlandIE63b] Sutherland, I.E. "SKETCHPAD: A Man-Machine Graphical Communication System", Spring Joint Computer Conference 1963, Spartan Books, Baltimore, Maryland, pp 326-ff. Graphical GUI drawing program, hardware uses early lightpen. Sketching system with snap-to-object, graphical constraints, recognition of graphical patterns (e.g. zig-zad, makes pretty). Cited in FoleyJD82 on stylus user-interface [SutherlandIE64b] Sutherland, Ivan E. "Sketchpad Demo (video) 1 of 2", Excerpt from early 1960's TV report, shown at SIGGRAPH '83: Fetched Generally credited as (almost) earliest direct-manipulation interface, often cites on gesture input [SutherlandIE64c] Sutherland, Ivan E. "Sketchpad Demo (video) 2 of 2", Excerpt from early 1960's TV report, shown at SIGGRAPH '83: Fetched Generally credited as (almost) earliest direct-manipulation interface, often cites on gesture input [SutherlandIE65a] Sutherland, Ivan E. "The Ultimate Display", Proc. IFIP congress, 1965, pp Essay on displays: not just visual, but haptic and physical output (e.g. force-feedback to a joystick), can simulate real physics, but also unreal physics: e.g. negative mass. Input can be not just keyboard or pointing (lightpen, RAND tablet), but sensing any body part, eye motion, etc. Cites to graphical constraints of Sketchpad, computer animation of Ken Knowlton. See also Alternate Reality Kit (literalism vs. magic) [SutherlandIE66a] Sutherland, I.E. "Computer Inputs and Outputs", Scientific American, September 1966 Overview article of Sutherland's "Sketchpad" system user-interface: describes the "interrupt" for I/O Describes Rand system, caret edit mark gesture to edit handwritten text, graphical editing Refers to sketchinput for prettying up drawings (user interface) Refers Ellis66/Ellis69 user-interface for flowcharts and handwriting recognition (see "further reading" in issue?) 8:26:10 PM]

26 ( ) History of Pen and Gesture Computing [SutherlandIE68a] Sutherland, I.E. "A head-mounted three-dimensional display", Proc. FJCC Fall Joint Computer Conference 1968 pp Binocular head-mounted display for simulating 3D virtual environments. Head position and orientation determined by four ultrasonic transducers, or electromechanical sensor. [SutherlandWR66b] Sutherland, William Robert "On-line Graphical Specification of Computer Procedures", Ph.D. Thesis, M.I.T, January Also MIT Lincoln Laboratories Technical Report 405, May 1966 Graphical programming languages / visual programming languages [TauschekG29] Tauschek, G. "Reading Machine", United States Patent 2,026,329, December 31, 1935 OCR Optical Character Recognition device using photo electric cell, and rotating drum with stencilled patterns for characters [Taylor54] Taylor "Indicating Apparatus", United States Patent 2,672,605, March 16, 1954 On a handwriting input terminal. Basically an X/Y electromagnetic digitizer with a grid of wires, but the output runs to ordinary electronics, not a computer. Grid wire encoding is not simple binary, but rather one-bit change in encoding from one wire to the next, to avoid bad signals from intermediate states (Gray coding). Cited in Irland64 [TeitelmanW63a] Teitelman, Warren "New Methods for Real-Time Recognition of Hand-Drawn Characters", Report 1015, Bolt Beranek and Newman, Cambridge, Massachusetts, June Also MIT Master's Thesis, Dept. of Math., 1962 Cited in Purcell77 [TeitelmanW63b] Teitelman, Warren "ARGUS - a program that recognizes hand-drawn characters in real time", Artificial Intelligence Project, MIT Computation Center Memo AIM-53, 1963 User manual for trainable handwriting recognition software described in "New Methods for Real-Time Recognition of Hand-Drawn Characters". Light-pen digitizer. Refers to construction of decision tree classifier. [TeitelmanW64a] Teitelman, W. "Real-time recognition of hand-drawn characters", AFIPS Proceedings of Fall Joint Computer Conference, Vol 26, Spartan Books, Baltimore, Maryland, pp , October 1964 Refers to putting constraints on recognition. Machine learning handwriting recognition using hypothesis testing. Criticizes invariants of Stylator system. Mentions recognizing mathematic expressions. [Teixeira68] Teixeira, J.K. and Sallen, R.P. "The Sylvania data tablet: a new approach to graphic data input", Proceedings of Spring Joint Computer Conference, pp , 1968 Cited in Rhyne86: Reference to early digitizer tablet, like RAND tablet, only transparent (touchscreen). [Telefunken68a] Bülow, Ralf "Auf den Spuren der deutschen Computermaus", Heise Online, April 28, 2009 "Rollkugel": essentially a mouse released with Telefunken TR-440 computer, two years prior to Engelbart mouse. [ThayerFW1878] Thayer, Frederick W. "Improvement in Masks", United States Patent 200,358, February 12, 1878 Original patent on baseball catcher's mask, showing heavy wire cage, goat-hair padding. [ThereminLS28a] Theremin, Leo Ssergejewitsch "Method of and Apparatus for the Generation of Sounds", United States Patent 1,661,058, February 28, 1928 Theremin: two-handed capacitive proximity sensors used to control pitch and volume of a musical instrument. Cited as early example of capacitive position sensing for touchscreens and proximity tablets. [ThomasRB67] Thomas, R.B. and Kassler, M. "Character recognition in context", Information Control, Vol 10, pp 43-64, 1967 Cited in DosterW77 [TinkerMA28a] Tinker, M.A. "The relative legibility of the letters, the digits, and of certain mathematical signs", Journal of Generative Psychology, Vol 1, 1928, pp (partial copy) Cited in Engel73 for early work on confusion/substitution errors in human recognition [TinkerMA63a] Tinker, M.A. "Legibility of print", Iowa, Iowa State University Press, 1963 Cited in IchikawaS84 : human recognition of characters [Tomita67] Tomita, Shingo, Noguchi, Shoichi, Oizumi, Juro, and Iwamoto, Kangi "Recognition of Handwritten Katakana Characters", Electronics Communications of Japan, Vol 50, 1967, pp OCR of handwritten recognition, features are line segments, quantized direction (chain) codes, types of joins/intersections, connection matrices [TownsendLG38a] Townsend, L.G. "Method of and Apparatus for the Indexing and Photo-Transcription of Records", United States Patent 2,121,061, June 21, 1938 Visually similar to Vannevar Bush "As We May Think" : microfilming of paper records to save space, combined with punch cards with serial number identifier for record, documents may be found by punched card, automatically displayed. [TurnerJA70a] Turner, J.A. and Ritchie, G.J. "Linear current division in resistive areas: Its application to computer graphics", Proc. SJCC, 1970, pp Perhaps the grand-daddy of resistive-sheet digitizer papers: Stylus injects a current into a point on a 8:26:10 PM]

27 ( ) History of Pen and Gesture Computing resistive sheet, sheet is grounded at two sides, current to the each side divides proportionally. Use isolation diodes for X and Y pairs of edges, some edge distortion due to point contacts of diodes. States 1% (of full scale) global accuracy, data rate up to 10,000 co-ordinates/second. [UhrL61a] Uhr, L. and Vossler, C. "A pattern-recognition system that generates, evaluates, and adjusts its own operators", Proceedings of World Joint Computer Conference, Vol 19, 1961, pp cited in O'Callaghan70, adaptive recognition (neural nets?) [VanRaalteJA68a] Van Raalte, John A. "Reflective liquid crystal television display", Proc. of IEEE, Vol 58 No 12, December 1969, pp (abstract only) Early reflective (non-backlit) LCD television display prototype, addressable in real time [VanderGon62] Van der Gon, J.J.D., et al "A handwriting simulator", Physics of Medicine and Biology, Vol 6 No 3, January 1962, pp Cited in Ehrich78 on modelling handwriting motion First author name also in print as van der Gon Denier [VanderGonDenier65] Van der Gon Denier, J.J. and Thuring, J. Ph. "The guiding of human writing movements", Biological Cybernetics, Vol 2 No 4, Feb 1965 pp First author name also in print as van der Gon Fast handwriting occurs without feedback (and is thus sloppy?), Shape is determined by muscle timing, not force [Vernon62] Vernon, H.C., and Walsh, Robert R. "Character Recognition Method and Apparatus", United States Patent 3,058,093, October 9, 1962 Zone-based character recognition, scanning intersections with zones [Viterbi67] Viterbi, A.J. "Error bounds for convolutional codes and an asymptotically optimal decoding algorithm", IEEE Trans. Information Theory, Vol IT-13, 1967, pp Cited everywhere (up until about 1980) for context correction / context in character recognition [VolkmannFC61a] Volkmann, Frances C. and Engen, Trygg "Three Types of Anchoring Effects in the Absolute Judgment of Hue", Jnl of Exp. Psychology, 1961, vol 61, No 1, pp (partial copy) Anchoring effect: factors that shift the classification / pattern recognition in human subjects. [VosslerCM64a] Vossler, C.M. and Branston, N.M. "The use of context for correcting garbled English text", Proceedings of 19th ACM National Conference, Philadelphia, 1964, pp D2.4-1 to D (abstract only) Just Abstract: see also DosterW77 context/dictionary to aid recognition: dictionary of English words containing probability of occurrence, and letter digram frequencies based on English text: See also Viterbi algorithm [WGBH64a] WGBH (Public Television Station) "Computer Sketchpad (Video)", fetched 2009 Video of Sketchpad from 1964: user interface with gestures and lightpen input. See video archives. [WaalJDD32a] Wall, J.D.D. "Printscript", Bibliography by T.R. Davis 1932 reference to unistroke writing styles? [WaltonHF1917a] Walton, Herman F. "Variable Electric Sign", United States Patent 1,231,821, July 3, 1917 Writing board using array of conductive bumps and a wired conductive stylus, as user writes on tablet, closes relay circuits to turn on lamps in matching array of lamps: can show electronic ink on sign. [WeizenbaumJ67a] Weizenbaum, Joseph "Contextual Understanding by Computers", CACM vol 10 no 8, August 1967, pp Eliza, early natural language understanding (typing text) simulating conversation. [West60] West, G.P. "Method and Apparatus for Sensing Handwritten or Printed Characters", United States Patent 2,964,734, December 13, 1960 Zone-based character recognition input, using electromechanical relays / logical sensing circuits [WeverEG28a] Wever, Ernest Glen and Zener, Karl Edward "The Method of Absolute Judgment in Psychophysics", Psych Rev, Vol 35, 1928, pp Differential sensitivity in judgements, rather than absolute: compare to pairwise-comparison method for character recognition of Pencept for handwriting, functional attribute method of Shillman et al [WhitneyFL67a] Whitney, F.L. "Electroluminescent Lamp", United States Patent 3,317,722, May 2, 1967 Electroluminescent (flexible) lamp, a.k.a. electroluminescent tape. [Wikipedia64] "PLATO (computer system)", fetched 3/22/2010 PLATO (Programmed Logic for Automated Teaching Operations), early educational computer system with custom terminal in 1964: bitmapped plasma graphics display and 16x16 low-resolution infrared touchscreen/touchpad input Project later did not involve custom hardware. [WilliamsTG69] Williams, Thomas G. "On-Line Parsing of Hand-Printed Mathematical Expressions: Final Report for Phase II", NASA Contractor Report CR-1455, Washington, D.C., December 1969, prepared by System Development Corporation Bernstein70, Martin71 cited this on user interface for mathematical 2-D parsing. User writes in two dimensions, compiler translates to linear form (shown at top of display) an evaluates. Characters are translated into recognized characters in two-dimensional form for editing in real time. Uses bounding rectangles in parsing. Gesture recognition described, handwriting recognition is in a separate publication. 8:26:10 PM]

28 ( ) History of Pen and Gesture Computing "Scrub mode" is a scratch-out gesture. Arrow gesture indicates move/edit. Horizontal line gesture for "open space/move" editing operation. Double horizontal line for close up. Dot (tap) plus second line to move a character. Two perpendicular stroke gesture for select/move group. See also Phase I and Phase IV reports. [Wilner66] Wilner "Dynamic alphanumeric Hand Printing Recognition System", IBM Technical Disclosure Bulletin, Vol 8, No 9, Feb 1966, pp Cited in Donahey76 zone handprinting/handwriting recognition system [WisemanNE69a] Wiseman, N.E. et al "PIXIE: A New Approach to Graphical Man-Machine Communication", Proc CAD Conf, IEE, Southampton 1969, pp 463-ff Early reference to pie menus: Cited in Pier 92 for earliest pop-up pie menus, might be a gesture? [WollrichAE70a] Wollrich, Arthur E. "Inductively Coupled Telautograph Apparatus with Stylus Angle Compensation", United States Patent 3,535,447, October 20, 1970 Stylus tilt correction in electromagnetic tablet/telautograph, two transmitter coils in stylus. No grid/matrix, sensor loops are all at periphery of tablet. Compare with Carau, Blesser for tilt correction? Cites to "scriptoscope", [WooPW62a] Woo, Paul W. "Ultrasonic Data converter", United States Patent 3,134,099, May 19, 1964 Acoustic digitizer, using an ultrasonic transducer in a stylus and linear arrays of microphones on the edges of the surface, measures time delay like sonar [WrightG52a] Wright, G.G.N. "The Writing of Arabic Numerals", Scottish Council for Research in Education Series No 33, pp 16-ff, University of London Press, London, 1952 (hardcopy book) Segmentation and variation on the digit 2, similar to chain codes Shows variability in writing styles for 2 and Z, confusion matrix: very interesting in demonstration handwriting variability on historical scale [YarbroughLD69a] Yarbrough, Lynn D. "Letter to the Editor: on using a graphic tablet", CACM, Vol 11 No 5, May 1968, pp. 294 and 377 Four-corner quadrilateral alignment/registration correction for a paper on a digitizing tablet: same calculation later applied for registration of tablet to display on a touchscreen. Example is a single correction for the entire paper, paper may be stretched (or photocopied) or tilted. [ZadehLA65a] Zadeh, L.A. "Fuzzy sets", Information and Control, Vol 8, 1965, pp Original paper on fuzzy sets, used for visual recognition/classification. Biswas81 cites on fuzzy sets for recognition. Electronic file includes 2012 PowerPoint presentation on fuzzy set theory. [ZernikeF34] Zernike, Fritz "Beugungstheorie des Schneideverfahrens und seiner Verbesserten Form, der Phasenkontrastmethode", Physica, Vol 1, 1934, pp Rotationally-invariant character recognition using polynomial moments. Zernike polynomials widely cited for optical recognition of certain patterns when viewed through circular apertures e.g. human eye, lens, radar image. Cited in Khotanzad90 Abstract: Auf Grundlage der Abb'eschen Beugungstheorie der optischen Abbildung wird das Aussehen eines Hohlspiegels mit willkürlich verlaufenden kleinen Abweichungen beim Foucaul tschen Schneideverfahren und beim neuen Phasenkontrastverfahren berechnet. Es werden die orthogonalen "Kreisfächenpolynome" gefunden und auf die Beugungserscheinungen beim kreisförmigen Spiegel angewendet. [Zobrak67] Zobrak, M.J. and Sze, T.W. "A method of recognition of hand drawn line patterns", Proceedings of 1st Princeton Conference on Information Sciences and Systems, 1967, pp handwriting recognition using eight-direction chain codes / direction sequences: other work mentioned for Chinese handwriting recognition. penhist (Prev page) Main Page (Next page) biblio75 "Everybody is ignorant, only on different subjects." -- Will Rogers, Vaudeville performer and political philosopher Einer der den Wald vor lauter Bäumen nicht sieht hat vielleicht auch eine besondere Fähigkeit, die verschiedensten Baumarten zu erkennen. 8:26:10 PM]

29 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1970 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:20 EDT Mirrors of this page can be found at: (Generally most up-to-date) biblio70 (Prev page) Main Page (Next page) biblio80 [ANSI72a] ANSI "Draft standard for ANSI74: Character Set for Hand-Printing", Draft Standard, American National Standards Institute, Inc, 1972 Draft for ANSI74 standard handwriting styles, with subset character sets [ANSI74a] ANSI "Character Set for Hand-Printing", American National Standards Institute, Incorporated, Standard X3.45, 1974 ANSI74 (ANSI X3.45) standard handwriting styles, with subset character sets [ARRL73] ARRL "Learning the International Morse Code", published by the American Radio Relay League, Newington, Connecticut, 1973 Shows preferred, rapid writing style for block-printed capital letters and digits, with stroke direction: taught by the US Military also, constrained handwriting for human recognition [AbbsJH71a] Abbs, J.H. and Sussman, H.M. "Neurophysiological Feature Detectors and Speech Perception: A Discussion of Theoretical Implications", Journal of Speech and Hearing Research, Vol 14, 1971, pp There are specific feature detection cells in human vision, and that there must be something similar for hearing: in any case, a good recognition system will be modelled on these feature extraction mechanisms. E.g. Vision has specific cells for horizontal and vertical lines. [AhlgrenRC71a] Ahlgren, R.C., Ryan, H.F. and Swonger, C.W. "A character recognition application of an iterative procedure for feature selection", IEEE Trans. Computers, Vol C-20, September 1971, pp (abstract only) Measure a number of features in a (typed character) dataset, then pick the features for recognition based on pairs of correlations [AhoAV75a] Aho, A.V. and Corasick, M.J. "Efficient String Matching: An Aid to Bibliographic Search", Communications of A.C.M., Vol 18, pp , 1975 String pattern matching [AlbanoA74] Albano, A. "Representation of digitized contours in terms of conic arcs and straight-line segments", Computer Graphics Image Processing, Vol 3, 1974, pp Constrained handwriting styles to make them readable by OCR 8:26:14 PM]

30 ( ) History of Pen and Gesture Computing [AmbrosioBF71a] Ambrosio, Blagio F. "Pressure-sensing Tablet", United States Patent 3,624,619, November 30, 1971 "pressure" resistive sheet tablet, two sheets held apart by thin web of rubber or other elastomeric material with many small perforations. Material is such that pressure from wrist or hand is not sufficient to make contact between sheets, but stylus point is. Assigned to Telautograph. [AndersonRA71a] Anderson, R. A. and Sibley, W. L. "Web Programming and Adaptive Heuristics", RAND Report R- 876-ARPA, September 1971 Web language (directed graph grammar): graphical programming (visual programming) language on a tablet could be transformed into resistor, capacitor, or other components, transformed by heuristics into higher-level logical objects. [AndersonRH72a] Anderson, Robert H. "Programming on a tablet: A proposal for a new notation", Proceedings of the symposium on Two-dimensional man-machine communication, pp Los Alamos, New Mexico. Sponsored by SIGPLAN : ACM Special Interest Group on Programming Languages Mentions use of two-dimensional tablet GUI for a programming interface for a graphical editor: actual article discusses a block-oriented programming language, somewhat similar to that of Plamondon from about Cites GRAIL (flowcharting, text); Anderson, Blackwell and Bernstein (mathematical notation); Datastructures (AMBIT) [Applicon73] Applicon Incorporated "Computerized Graphic Processing System: System User's Manual", Applicon Incorporated, September 1973 Sketch recognition / world's first gesture recognition user-interface in a commercial product: describes Ledeen gesture recognizer? [AshfordRF72a] Ashford, Richard F. Jr "RADIK: An Interactive Graphics and Text Editor", Master's thesis, US Naval Postgraduate School, Moneterey CA, June 1972 XDS 9300 computer with ADAGE AGT/10 graphics computer, light-pen input, basic graphics operations (rotation/translation/scaling of drawings, etc.) [BBN75a] BBN Bolt, Beranek and Newman "Interface Message Processors for the ARPA Computer Network", BBN Report 2988, Quarterly Technical Report 8, 1 October 1974 to 31 December 1974 Deployment of IMP processors for ArpaNet: debugging of single IMPs in simulation works well, debugging in cluster not well. Code checksummed periodically for reliability. Multiple redundant processors easy to debug, system continues operation even with some wires pulled out, etc. Case-analysis of network surviving partial failures. Compare with high-availability systems? [BaerJA72a] Baer, James A. and Hill, John W. "Optical-to-Tactile Image Conversion for the Blind", Stanford Research Institute Project Report, Projects 1417 and 8647, 30-June-1972 Optacon touch sensor for the blind: small set of vibrating haptic transducers with hand-held scanner. Output to one or two fingers. Intended as accessibility aid for reading of printed material. [BarkanH73a] Barkan, Harold; Barkan, Edward D.; and Swartz, Jerome "Transparent Touch Controlled Interface With Interactively Related Display", United States Patent 3,757,322, September 4, 1973 Transparent digitizer touchscreen using transparent switches on a display. Mentions need to mount closely to display to minimize visual parallax so that combination appears to user as a single device. Note: multitouch -- "child touches the clock face" and a second image concurrently. Small pads for use with stylus for high resolution (contact density). Cites to Johnson [BatchelorBG74] Batchelor, B.G. "Practical Approach to Pattern Classification", Plenum Press, London/New York, 1974 Chapter 2: Contains examples of boundary theory decision surface diagrams, but doesn't discuss it that way. Refers to human "teacher" for trainable recognition: e.g. perceptual/cognitive ambiguity about what is an "A" vs an "H". Decision surfaces in multi-dimensional feature analysis (pair-wise comparison?). Statistics: similarity, distance, and compactness of decision surface/classification. Implies boundary theory vs category theory. Accidentally shows European variants of hand-written "H" character [BatterJJ71a] Batter, James J. and Brooks, Frederick P. jr. "GROPE-1: A Computer Display to the Sense of Feel", Proc. IFIP Congress, 1971, pp Two-axis / two-dimensional haptic force-feedback display, used to illustrate to students electromagnetic and gravitational fields in two dimensions. [BeckF73a] Beck, F. and Stumpe, B. "Two Devices for Operator Interaction in the Central Control of the New CERN Accelerator", CERN report 73-6, Geneva, 24 May 1973 Transparent capacitive touchscreen over CRT display. Copper on glass is "invisible". Approach of finger changes capacitance to ground. See also Stumpe. [BeckerPW72] Becker, Peter W. and Nielsen, K.A. "Pattern recognition using dynamic pictorial information", IEEE Trans. Systems, Man, and Cybernetics, July 1972, pp Book review of 1974 reference: use of context, frequency of occurrence of binary words (two characters / character pairs?) in character recognition. 8:26:14 PM]

31 ( ) History of Pen and Gesture Computing [BeckerPW74] Becker, Peter W. "Recognition of Patterns Using the Frequencies of Occurrence of Binary Words", Springer Verlag, New York and Vienna, 2nd edition, 1974 (abstract only) Book edition of doctoral thesis on combining subpatterns that are classified/recognized less reliably to get a better overall recognition performance [BennetJR75a] Bennet, J.R. and McDonald, J.S. "On the measurement of curvature in a quantized environment", IEEE Trans. Computers, Vol C-24, 1975, pp [BensonB73a] Benson, Bengt Anders "Perception Apparatus for the Blind", United States Patent 3,740,446, June 19, 1973 Tactile/optical scanner for the blind. Cites to Optacon. Mechanical spiral scanner, display "pens" (similar to braille display) show dark/light and dark/light transitions, e.g. edges. [BernsteinMI71a] Bernstein, Morton I. "Computer Input Output of Two-Dimensional Notations", Proceedings of Second Symposium on Symbolic and Algebraic Manipulation, Los Angeles, California, March 23-25, 1971 Review/opinion article on mathematical expression input and output using tablet. Cites to GRAIL, etc. [BernsteinMI75a] Bernstein, M.I. "Interactive Systems Research: Interim Report for the Period 16 September 1974 to 15 March, 1975", System Development Corporation, Santa Monica, California, Report No SDC-TM-5243/003/00, 15 May, 1975 NTIS citation index Speech understanding system, describes acoustic-phonetic and lexical-mapping processes [BernsteinMI75b] Bernstein, M.I. "Interactive Systems Research: Final Report to the Director, Advanced Research Projects Agency, for the Period 16 September 1974 to 15 September, 1975", System Development Corporation, Santa Monica, California, Report No SDC-TM-5243/004/00, 15 November, 1975 NTIS citation index Speech understanding system, describes acoustic-phonetic and lexical-mapping processes [BerthodM74] Berthod, M. and Maroy, J.P. "Morphological Features and Sequential Information in Real-time", Proceedings of the 2nd International Joint Conference on Pattern Recognition, August 1974, pp Refers to Blesser, et al and functional attributes (distinguishing features). Mentions use of pressure (actually, force) and timing (!) in handwriting recognition -- very gesture-like. [BlesserB72a] Blesser, B., and Ozonoff, D. "A Model for the Radiologic Process", Radiology, Vol 103 No 3, pp , June, 1972 Barry's paper on perception in reading medical X-rays. Set error leading to recognition/perception bias: you found what you are looking for, so you do not recognize other, fairly obvious things. Plausible interpretation == grammar of seeing, what is a legitimate perceptual "sentence". [BlesserB73a] Blesser, B., Shillman, R., Cox, C., Kuklinski, T., Ventura, J., and Eden, M. "Character Recognition Based on Phenomenological Attributes", Visible Language, Vol 7 No 3, 1983, pp Examples of attributes/features: functional closure, perceptual closure, physical closure [BlesserB73b] Blesser, B., Shillman, R., Kuklinski, T., Cox, C., Eden, M. and Ventra, J. "A Theoretical Approach to Character Recognition Based on Phenomenological Attributes", Proceedings of the 1st International Joint Conference on Pattern Recognition, 1973, pp Published version of Shillman's Thesis work: pair-wise distinguishing characteristics (features) of handprinted characters. [BlesserB74a] Blesser, B., Shillman, R., Kuklinski, T., Cox, C., Eden, M., and Ventura, J. "A Theoretical Approach for Character Recognition Based on Phenomenological Attributes", International Journal of Man-Machine Studies, Vol 6, pp Character recognition based on ambiguously shaped characters or "difficult cases" of low quality Graphical context rules extracted via psychological experimentation Arguments for boundary-theory pair-wise comparison for recognition Contains chart of Shillman's ambiguously shaped characters [BoumaH71a] Bouma, H. "Visual recognition of isolated lower case letters", Vision Research, Vol 11, 1971, pp Cited in Milloy78 for confusion/substitution errors in human recognition for lower-case characters Cited in Bozinovic89 for human recognition by outline of word (as shown in WrightG52) [BremermannHJ71] Bremermann, H.J. "What Mathematics Can and Cannot Do for Pattern Recognition", in "Zeichenerkennung durch biologische und technische Systeme", Proceedings of the 4th Congress of the Deutsche Gesellschaft für Kybernetik, Berlin, pp Published by Springer-Verlag, New York, April 6-9, 1971 Fundamental knowledge of the brain would greatly help on pattern recognition by machine Difficulties in pattern recognition are generally combinatorial Most theoretical papers on pattern recognition are quite worthless Problems like handwritten character recognition are not even well-defined Handwritten "A"s are not mathematically defined: just what people agree on Qualitative and formal-language "theorems" do not help solve pattern recognition problems Pattern recognition theorems (perceptron/neural-net) working without quantitative combinatorial analysis Adaptive systems may not converge on optimum performance: stick on local maxima Most papers give no reasons for choice of recognition features Most recognition 8:26:14 PM]

32 ( ) History of Pen and Gesture Computing systems fail on choice of features Features chosen must separate patterns in recognition Humans do not use mathematical features, like string probabilities Gives fundamental reasons why polynomial mathematical features will not work Cybernetic recognition (measure deformation from paradigm) vs fuzzy sets vs perceptrons, etc. Steepest descent, other gradient methods in statistical pattern recognition fail due to local minima Picture grammars (linguistic scene descriptions) useless if elements cannot be identified Creative imagination of researchers better than statistical recognition Examples of nature best design inspiration in recognition Many papers just an exercise in fashionable recognition techniques "mount Bayes is overpopulated" -- critique of statistical pattern recognition Feature extraction plays a role in visual cortex [BridleJS73] Bridle, J.S. "An efficient elastic-template method for detecting given words in running speech", British Acoustical Society Spring Meeting, April 1973, Chelsea College, London, Paper 73SHC3 Cited in Kruskal83 [BurckhardtCW71] Burckhardt, Christof W., Levrat, Bernard, and Perrin, Marie-Christine "Character Recognition with Incremental Rate Encoding", Kybernetik, Vol 8 No 5, 1971, pp , Germany. Handwriting recognition using acceleration chain codes as the feature (?) [BurtonRP73a] Burton, Robert Preece "Real-time measurement of multiple three-dimensional positions", PH.D. Thesis, Department of Computer Science, U. Utah, 1973; also Tech. Report UTEC-CSc , June D three-dimensional digitizer for multiple points. Points are light-emitting diodes LEDs. The light-emitting diodes are easily distinguished from one another by flashing them on and off in sequence, under computer control. The position of a light source is measured by scanning the environment. Fast one-dimensional, geometric scans quickly determine planes in which a light source lies. The point of interest is at the intersection of these planes. [BurtonRP74a] Burton, Robert P. and Sutherland, Ivan E. "Twinkle-box: a three-dimensional computer input device", AFIPS Joint Computer Conference, May , Chicago, pp Cites to Lincoln Wand, Lawrence G. Robert, 1963, three-dimensional acoustic/sonic digitizer: this design is an optical digitizer in three dimensions [Caelli75] Caelli, Terrence M. and Finlay, David C. "Pattern discrimination of Gaussian distributions of orientation and line length parameters", Perception and Psychophysics, 1975, Vol 16 No 3, pp From Kuklinski file [Caskey72] Caskey, David L. and Coates, C.L. "Machine Recognition of Handprinted Characters", Electronics Research Center, University of Texas at Austin, Technical Report No 126, May 1, 1972 (hardcopy book) Broad (1972) review of problems handwriting recognition must deal with: consistent performance (gross features only are used), etc. Has many pages of samples of the handwriting collection they used [Caskey73] Caskey, D.L. and Coates, C.L. Jr "Machine recognition of handprinted characters", Proceedings of the 1st International Joint Conference on Pattern Recognition, November 1973, pp handwriting recognition on for writing the ANSI standard characters: constraints [CataldoJW74] Cataldo, J.W. "Lettering", Davis Publications, Worcester Massachusetts, 1974 (hardcopy book) Many examples of calligraphy, hand-lettering, embellishments on basic hand-written letter forms, including stroke directions, hooks, etc. Handwriting variability. [CencelJA73a] Cencel, J. Arthur "Capacitive Keyboard", United States Patent 3,750,113, July 31, 1973 Capacitive keyboard, keys wired into matrix of rows and columns. Individual keys are read by addressing rows, then columns, and the state read into a shift register where the index in the shift register matches the row/column address of the key. Thus, all keys can be addressed and read independently for N-key / whole keyboard rollover. Compare with Kaplow multi-touch [CerfVG74a] Cerf, Vinton G. and Kahn Robert E. "A Protocol For Packet Network Intercommunication", IEEE Trans on Communications, Vol Com-22 No 5, May 1974, pp Early packet-switching-network paper describing TCP/IP (pre-udp), including varying packet sizes, end-toend error checking, tolerance of transmission failures. [ChandrasekaranB71] Chandrasekaran, B. "Independence of measurements and the mean recognition accuracy", IEEE Trans. Information Theory, Vol IT-17, 1971, pp Corrections in Vol IT-19, 1972 Martin,GL90 on analysis of how well neural nets work [ChangCY73] Chang, C.Y. "Dynamic programming as applied to feature subset selection in pattern recognition systems", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-3, March 1973, pp Cited in Fisher75 [ChowC75a] Chow, C. "Effect of Line Width on recognition of V's and Y's", S.B. Thesis, Department of Electrical Engineering and Computer Science, August 1977, Massachusetts Institute of Technology, Cambridge, Massachusetts. Massachusetts Institute of Technology undergrad Thesis on line width and functional attributes [ChristensenC71a] Christensen, Carlos; Wolfberg, Michael S.; and Fischer, Michael J. "A Report on AMBIT/G, Vols 1-4", Applied Data Research Inc. Report CA , February 26, 1971 Final report on project "Machine-Independent Software Programming" AMBIT. Visual programming 8:26:14 PM]

33 ( ) History of Pen and Gesture Computing language, touchscreen graphical editing with gestures, running on TX-2 computer. References to Ledeen recognizer, constraint languages. agd symbolic debugger. See additional materials in files. [ChristensenC71b] Christensen, Carlos "An Introduction to AMBIT/L, A Diagrammatic Language for List Processing, Third Edition", Applied Data Research Inc. Report CA , February 22, 1971 Implementation of visual programming language AMBIT/G, represented as two-dimensional directed-graph diagrams for both data and rules. [ChristensenC71c] Christensen, Carlos and Wolfberg, Michael S. "AMBIT/G as an Implementation Language", Applied Data Research Inc. Report CA , March 2, 1971 Visual programming language AMBIT/G, requires definition of all data structures/representations before starting programming algorithms. [ClarkRK71a] Clark, R.K. "Identification System", United States Patent 3,621,720, November 23, 1971 Cited in Herbst76 signature verification stylus using tip force/pressure vs. time [Context73] Context Corporation "OCR in Word Processing - a guide for the word processing supervisor", 9 Ray Avenue, Burlington, Massachusetts 01803, 1973 [CooperLN73a] Cooper, Leon N. "A Possible Organization of Animal Memory and Learning", Collective Properties of Physical Systems, Nobel 24 (1973), pp Brief talk on Leon Cooper on getting interested in neural networks and memory, after getting Nobel prize re superconductivity. Founder of Nestor handwriting recognition company. [Corballis75] Corballis, M.C. and Roldan, C.E. "Detection of symmetry as a function of angular orientation", Journal of Experimental Psychology: Human Perception and Performance, Vol 1, 1975, pp Cited in IchikawaS84 Angle (vertical vs horizontal) perception in humans? [CormackEO74] Cormack, Elizabeth O. and Robert H. "Stimulus configuration and line orientation in the horizontal-vertical illusion", Perception and Psychophysics, Vol 16 No 2, 1984, pp Context of adjacent lines in human perception of vertical and horizontal line length: human feature perception (many other references on this topic) [CottonI72] Cotton, I. "Network graphic attention handling", Online 72 International Conference, Brunel University, Uxbridge, England September 1972, pp using keyboard, shaft encoder, other devices as virtual replacements for lightpen/tablet/mouse. finger as stylus, Lincoln Wand as 3D digitizer. Cited in Wallace76 for "virtual devices" [CoxC74] Cox, C., Blesser, B., Eden, M. "The Application of Type Font Analysis to Automatic Character Recognition", Proceedings of the 2nd International Joint Conference on Pattern Recognition, Lyngby, Denmark, Aug 13-15, 1974 OCR: Variability is two sources: type font, and scanner/ink effects. Most variability among characters describable in terms of stylistic consistency. Stated aim of pre-processing is to reduce variability recognizer must deal with. Adaptive recognition should analyze for stylistic consistency as a separate variable, and separate variability from recognition. Variability model is NOT same as picture grammar approach. [CraneHD75a] Crane, Hewitt David, and Savoie, Robert Ellis "Handwriting System", United States Patent 3,930,229, December 30, 1975, assigned to Stanford Research Institute, Menlo Park, California. SRI/Crane's accelerometer-pen character recognition: Uses ULDR chain codes, similar to BLRT chain codes [CraneHD75b] Crane, Hewitt David and Savoie, Robert Ellis "Special Pen and System for Handwriting Recognition", United States Patent 3,906,444, September 16, 1975, assigned to Stanford Research Institute, Menlo Park, California. SRI/Crane's accelerometer-pen for signature verification/recognition e [CummingsTF73a] Cummings, T.F. "Variable Keyboard for Terminal Displays", IBM Technical Disclosure Bulletin, July, 1973, pp On-screen/virtual keyboard with labels displayed on monitor, transparent plastic keys over the display. Keyboard functions change with application. (On-screen menus) [DarringerJA75a] Darringer, J.A. et al "Speed Pen", IBM Technical Disclosure Bulletin, Vol 18 No 7, December 1975, pp Acoustic digitizer: horizontal raised lines across surface, piezo transducers or microphones pick up sound of ballpoint pen's vertical movement over lines while user writing signature. Biometric verification of signature based on Y velocity profile. [DayAM72] Day, A.M., Parks, J.R. and Pobgee, P.J. "On-line written input to computers", in "Machine Perception of Pictures and Patterns", The Institute of Physics, London, England, 1972, pp Also cited as Proceedings of the Conference on Machine Perception of Patterns: Tedding England. Claims 2% error rate (but 8.5% reject): feedback to users greatly improves handwriting recognition Handwriting recognition features are coding zones, sequence similar to chain codes, number and length of horizontal/vertical strokes Describes resistive-sheet/film digitizer tablet, one character big: user-interface for handwriting, one-character input [DeQuervainA75a] de Quervain, Alfred and Wild, Peter "Information-bearing Devices and Projection Display 8:26:14 PM]

34 ( ) History of Pen and Gesture Computing Systems Therefor", United States Patent 3,895,866, July 22, 1975 Early LCD projector patent. Static information can be printed on the glass as well. [DertouzosML72a] Dertousoz, M.L. "Graphic Data Tablet", United States Patent 3,705,956, assigned to Computek, Incorporated, Cambridge, Massachusetts, December 12, 1972 Early electromagnetic digitizer tablet patent, using Gray Code grid pattern encoding, using phase of induced voltage [DudaRO72a] Duda, R.O and Hart, P.E. "Use of the Hough transform to detect lines and curves in pictures", Graphics and Image Processing, Vol 15, pp 11-15, 1972 Finding/detecting lines in B/W pictures: example is edges of a machine cabinet, extracts parameters first, lines detected indirectly. Cited in Kahan87 regarding a statistical Bayesian classifier using binary features. [Dyche72] Dyche, James W. "Handwriting Authentication Technique", United States Patent 3,699,517, assigned to Sylvania Electric Products Inc., Cotber 17, 1972 Herbst76 cites this on signature verification using force, velocity, pen lifts, correlationsamong them Signature verification using velocity and acceleration, using a graphic tablet [Ehrich73] Ehrich, Roger W. "A Contextual Post-Processor for Cursive Script Recognition - Summary", Proc. 1st Int. Joint Conf. Pattern Recognition, Oct. 1973, pp Cited by Marlin Eller, Microsoft Pen Computing group [Ehrich75] Ehrich, Roger W. and Koehler, K.J. "Experiments in the Contextual Recognition of Cursive Script", IEEE Trans. Computers, Vol C-24 No 2, February 1975, pp Contextual pre-processor for script using letter length, word context, stroke parsing: best character alternatives given to word-context recognizer (likelihood ratio) Due to scope of problem, one becomes mired in programming 1.3% error when training and test data were identical Word-wise recognition of hand-written script Refers to certain degenerated, trivial, toy recognition problems and their pointlessness Need for low-quality data in training a system Goshtasby88 cites for binary letter pairs (digrams) for spelling context, following by spelling dictionary look-up for what's left [Eichen73] Eichen, Howard R. and Paulsen, Robert H. "Processing of Coordinate Values of Points Entered in a Graph Board or the like", United States Patent 3,720,948, assigned to Co-Data Corporation, Fullerton California, March 13, 1973 Describes dividing a tablet surface into a simulated keyboard area, a sketch area, virtual tablets, etc. Relevant to Schumer patent [ElectronicsReview71] Electronics Review "Program Recognizes Handwritten Math", Electronics Review, Vol 44, No 18 (1971) Cited in Sklarew99 patent [Engel73] Engel, G.R., Dougherty, W.G., and Jones, G.B. "Correlation and Letter Recognition", Canadian Journal of Psychology, 1973, Vol 27 No 3, pp Mentions response bias in measuring human recognition errors Problems of matching recognition results of tests and predictions based on only one variable: human recognition is a correlation function of multiple variables Points out problems of weighting multiple variables/features [Feng75] Feng, H.Y. and Pavlidis, T. "Decomposition of polygon into simpler components: Feature generation for syntactic pattern recognition", IEEE Trans. Computers, Vol C-24, Jun 1975, pp Cited in Agui79 [FernaldOH73a] Fernald, Olaf H. "Optical graphic data tablet", United States Patent 3,761,877, September 25, 1973 Cited in Tappert's patent list of 1986 on on-line/dynamic handwriting recognition Cited in Fox88: on annotation drawing/handwriting Optical digitizing tablet: stylus shines a light on a transparent glass plate, with a scanner underneath [Fisher75] Fisher, Edward G., Riseman, Edward M., and Hanson, Allen R. "Feature Selection Using Non-redundant Thresholded Measures", preliminary manuscript, University of Massachusetts, Amherst, Massachusetts, 1975 Pair-wise discrimination: guidelines for when to work harder on hard-to-separate classification pairs [Fitzgerald75] Fitzgerald et al "Procedure for rectifying a roughly sketched drawing", IBM Technical Disclosure Bulletin, Vol 18, No 7, December 1975, pages Cited in Fox92 [FitzhughRJ71a] Fitzhugh, Robert J. and Katsuki, David "The touch-sensitive screen as a flexible response device in CAI and behavioral research", Behavioral Research Methods and Instrumentation, Vol 3 No 3, 1971, pp Translucent (transparent) touchscreen with rear-projection, using X/Y grid of conductors with physical contact of an X and a Y wire. Mentions multiple touches (multi-touch) as something to be detected and rejected. Did not find a transparent and flexible conductive material: polacoat is conductive and transparent, but not flexible. Metallic-coated mylar for resistive/conductive film touchscreen. [FodorJA74] Fodor, J.A., Bever, T.G., and Garrett, M.F. "The Psychology of Language", McGraw-Hill, New York, 8:26:14 PM]

35 ( ) History of Pen and Gesture Computing 1974 Adjustment of perception of stimulus to be what is "reasonable": We accept hypotheses about the identity of individual letters and words only if they yield a sensible interpretation of longer sequences like phrases and sentences. (Context information in recognition) [FoleyD71] Foley, D. "The Probability of Error on the Design Set as a Function of the Sample Size and Feature Size", PhD dissertation, Syracuse University, June 1971 Cited in Meisel: 85 [FoleyJD74] Foley, J.D. and Wallace, V.L. "The Art of Natural Graphic Man-Machine Conversation", Proc. IEEE, Vol 62 No 4, April 1974, pp (partial copy) Cited in FoleyJD84 Cited in Wallace76 for "virtual devices" (virtual tablet) [ForneyGE73a] Forney, G.E. Jr. "The Viterbi algorithm", Proceedings of the IEEE, Vol 61, 1973, pp Cited in Sinha88, Hull83a Description of Viterbi algorithm, applications to text/character/ocr recognition using context. [FreedmanMD74a] Freedman, M.D. "Optical character recognition", IEEE Spectrum, Vol 11 No 3, March 1974, pp Cited in Nouh84 [FreemanH74a] Freeman, Herbert "Computer processing of line-drawing images", Computing Surveys, Vol 6, 1974, pp Long treatment of art on extracting line drawings / line structures (line traces) from images of line drawings, and also on semantic understanding of line structures. Cites to terrain recognition from topological map data. Cited in Kerrick88 [FuKS74] Fu, K.S. "Syntactic Methods in Pattern Recognition", New York: Academic Press, 1974 Cited in Baird84 [FujisakiH71] Fujisaki, H., Nagai, S. and Hidaka, N. "On-line recognition of hand-written numerals", Annual Report of the Engineering Research Institute, Faculty of Engineering, University of Tokyo, Japan, Vol 30, August 1971, pp Tappert's bibliography [GoldsteinIP74a] Goldstein, Ira P. "Summary of MYCROFT: A System for Understanding Simple Picture Programs", Artificial Intelligence, Vol 6 No 3, 1975, pp , also M.I.T. AI report 305, Logo report 10, May 1974 Program to repair distorted line drawings (not images) automatically. Pitch: Compare with later graphical constraints, constraint-based programming, graphical syntactic rules, Waltz 1970's on picture understanding. [GoldsteinIP74b] Goldstein, Ira P. "Understanding Simple Picture Programs", Technical Report 294, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, 1974 Earlier, more detailed presentation of other Goldstein 74. Drawing program uses Turtle graphics, planner creates program. [Goodwin75] Goodwin, N.C. "Cursor Positioning on an Electronic Display Using Light-pen, Light-gun, or Keyboard for Three Basic Tasks", Human Factors, pp , Vol 17 No 3, 1975 Study of positioning time with various cursor, pen, keyboard devices [GranlundGH72a] Granlund, G.H. "Fourier preprocessing for hand print character recognition", IEEE Trans. Computers, Vol C-21, pp , 1972 [Granzotti73] Granzotti, A.N. "Free Hand Drawing Display System Utilizing Light Pen to Write and Erase Information", United States Patent 3,758,717, September 11, 1973, assigned to International Business Machines Corporation, Armonk, New York. User-interface: Hardware patent on digitizer erasing on display with a light pen, makes broad claims on erasure/editing of freehand writing or sketching of electronic ink [Grether73] Grether, C.B. and Stroh, R.W. "Subjective Evaluation of Differential Pulse-Code Modulation Using the Speech 'Goodness' Rating Scale", IEEE Trans. Audio and Electroacoustics, Vol AU-21 No 3, June 1973, pp "goodness" scale for subjective evaluation of speech quality: relates to cognitive psychology experiments to extract functional attributes Method for subjective evaluation of cognitive/psychophysical perception/recognition, where theory is not adequately developed for quantitative prediction [Gries71] Gries, D. "Compiler Construction for Digital Computers", Wiley & Sons, New York, 1971 A standard reference on compiler /programming language design [GuentherJA73a] Guenther, Joel Arthur "Machine recognition of hand-sent Morse code using the PDP-12 computer", Air Force Institute of Technology Report AD , December 1973 Master's Thesis at USAF Inst. of Tech. Recognizer for human-sent Morse code, 95% accuracy, biggest problems is detecting the tone in the presents of noise (sampling) [HallRE72] Hall, R.E. and Hulbert, L.N. "Machine Recognition of Symbols", United States Patent 3,676,848, July 11, 1972, assigned to Ferroxcube Corporation, Saugerties, New York. 8:26:14 PM]

36 ( ) History of Pen and Gesture Computing Early patent on zone/region on-line character recognition [Hanson73] Hanson, A.R. and Riseman, E.M. "System Design of an Integrated Pattern Recognition System", Proceedings of the 1st International Conference on Pattern Recognition, Washington, D.C, October 30 - November 1, 1973 (abstract only). COINS Technical Report 73C-5, University of Massachusetts, Amherst, Massachusetts, June 1973 Cited in Fisher75 [HansonAR74a] Hanson, A.R., Riseman, E.M., and Fisher, E. "Context in Word Recognition", COINS Technical Report 74C-6, August 1974 Cited in DosterW77 [HarmonLD72] Harmon, L.D. "Automatic Recognition of Print and Script", Proceedings of the IEEE, Vol 60, 1972, pp Cited in Litvin82, Nouh84 [HarrisTJ73a] Harris, Thomas J. "Modular touch sensitive indicating panels with touch and light transmissive overlay cover membrane containing visible printed indicia", United States Patent 3,777,222, December 4, 1973 Transparent touchpanel using grid of conductors, may be overlaid on printed indicia display. Modular construction as component for different application systems. [HeartFE73a] Heart, Frank E. "Interface Message Processors for the ARPA Computer Network", Bolt Beranek and Newman Quarterly Technical Report No.3, Report No. 2667, AD , October 1973 Technical report on IMP/TIP interface message processors / terminal input processors for ARPANETY (pre- Internet). Data performance files transferred over network, instead of tape. Incremental/continuing changes to routing tables, routing based on dynamic connectivity/performance. [Heinz72] Heinz, B., Paprottka, H., and Schulte-Derne, M. "Meine liebe Fibel: Schreiblehrgang", Verlag Ferdinand Kamp Bochum, 1972 German/European handwriting style textbook [Hemstreet73] Hemstreet, Harold S. "Video Amplifier", United States Patent 3,713,100, January 23, 1973 Cited in Bokser88; Cited in Tappert88b: on segmentation combined with character recognition Optical/typed character recognition by comparing with a set of individual prototypes for each possible shape, and taking best match score. Comparison may be pixel-wise or slice-wise comparison. [HenckelsLP71] Henckels, L.P. "A Parametric Method for Developing and Evaluating Handprinted Character Recognition Algorithms", ScD Thesis, Department of Electrical Engineering, Massachusetts Institute of Technology, June 1971 Massachusetts Institute of Technology Thesis on generating simulated/artificial variability in handwriting Find samples at the boundary of where recognition problems occur Says problems in recognition algorithms should be found systematically Says need 10,000 training samples to get feature space error down to 1% Says Neisser's figure for human recognition too poor for practical device(!) Says testing takes up most of the effort, but is treated too lightly Quotes "dominant factor factor in performance is care in printing" [HenrichonEG75] Henrichon, E.G. and Bloom, H.J. "Method and System for Optical Character Recognition", United States Patent 3,930,231, December 30, 1975, assigned to Xicom Data Entry Corporation, Newton Upper Falls, Massachusetts. Three methods of character recognition are matrix matching, Feature extraction, and curve tracing [HerotCF74a] Herot, Christopher F. "Using Context in Sketch Recognition", Master's Dissertation, Massachusetts Institute of Technology, Cambridge, Massachusetts, 1974 Context in sketch recognition (sketch editing?) on tablet. Example is syntactic rules for construction of a house: e.g. house/living-room/wall/window [HerrickEM74] Herrick, E.M. "A Taxonomy of Alphabets and Scripts", Visible Language, Vol VIII No 1, Winter 1974, pp 5-32 [HitachiI75] Hitachi, I. "Direct Input System for Handprinted Programs and Data", Central Research Laboratory of Hitachi, Ltd., Tokyo, Japan, 1975 Cited in Purcell77 [Hlady75] Hlady, A.M. "Touch Sensitive Position Encoder using a Layered Sheet", United States Patent 3,916,099, October 28, 1975, assigned to Canadian Patent and Development Limited, Ottowa, Canada. Digitizer using surface acoustic wave: transparent touchscreen [HoltAW74] Holt, Arthur W. "Algorithm for a Low Cost Hand Print Reader", Computer Design, February 1974, pp Simple OCR for handwriting recognition, using a single vertical line as a constraint, measuring closed/open loops and areas as features [Honeywell75a] Honeywell Information Systems "MULTICS Programmer's Manual Reference Guide, Series 60 (Level 68)", Honeywell Information Systems AG91-01 Dec Multics operating system architecture and overview: CLI command language, input/output facilities, file 8:26:14 PM]

37 ( ) History of Pen and Gesture Computing system. Segmentation/paging, virtual memory; access control lists for selective sharing (ACL), access isolation mechanism (AIM), ring structure and levels of privilege. Introduction to Multics software development environment, dynamic (run-time) linking to subroutines. Unstructured, sequential, indexed files. Section VI on access control, modes, syntax of ACLs. AIM allows delegation of access privileges grantor has via ACL. Processes (and all objects) identified by name Project.User.Tag. See ContentGuard Google. Automated full and incremental backups (dumps) of files/objects. Access class associated with an object for its lifetime. [HongJP74] Hong, J.P. "Pattern Recognition: Invariant Stochastic Feature Extraction and Statistical Classification", in JPL Space Programs Summary 37-58, Vol III, 1974 Character recognition features used by IBM OCR page reader. Measurements on data (feature extraction) and classification algorithm closely tied to problem at hand [HooSK75] Hoo, S.K. "Coordinate Locator", IBM Technical Disclosure Bulletin, Vol. 18 No. 5, October 1975 pages Cited in Bennett 91 optical stylus digitizer tablet somewhat similar to Anoto: visually transparent surface using a coded grid placed over a display: appears to give proximity detection also. [Howells75] Howells, J.A. "Apparatus for Recognizing Hand Printed Characters", United States Patent 3,909,785, September 30, 1975, assigned to Amperex Electronics Corporation, Hicksville, New York. Nine-zone/sector/region recognition [HurstGS72a] Hurst, George S. "Electrical Sensor of Plane Coordinates", United States Patent 3,662,105, May 9, 1972 Resistive sheet digitizer, two resistive sheets with AC voltage gradients in orthogonal directions. Single probe (not stylus) punctures both sheets and picks up both signals. [HurstGS74a] Hurst, George S. "Electrographic Sensor for Determining Planar Coordinates", United States Patent 3,798,370, March 19, 1974 Resistive sheet digitizer, single sheet, voltage gradient applied alternately in vertical and horizontal direction, using second conductive sheet and intermediate layer of deformable insulation such as a gel, that permits the two sheets to be pressed into effective contact. [HurstGS75a] Hurst, George S. and Colwell, William C. Jr. "Discriminating Contact Sensor", United States Patent 3,911,215, October 7, 1975 Resistive sheet digitizer, single sheet, voltage gradient applied alternately in vertical and horizontal direction, using second conductive sheet and small spacer bumps to hold two sheets apart: palm discrimination versus narrow stylus tip. [HussainABS72a] Hussain, A.B.S., Toussaint, G.T., and Donaldson, R.W. "Results Obtained Using a Simple Character Recognition Procedure on Munson's Handprinted Data", IEEE Trans. Computers, Vol 21, February 1972, pp (abstract only) Refers to Munson's data collection / test sample data base for handwriting [IJCPR74] IJCPR Program "Second International Joint Conference on Pattern Recognition", Lyngby, Denmark, August 1974 [IannouJT75] Iannou, J.T. "Digital Position Measurement System with Stylus Tilt Error Compensation", United States Patent 3,873,770, March 25, 1975, assigned to Bendix Corporation, Southfield, Michigan. Tilt correction patent for tablet digitizer Tappert's patent list shows this as 3,875,770. [IchikawaT74] Ichikawa, T. and Yoshida, Y. "On-line Recognition of Handprinted Characters with Associative Read-Out of Patterns in Memory", Proceedings of 2nd International Joint Conference on Pattern Recognition, August, 1974, pp Cited in Litvin82 [IijimaT72] Iijima, Taizo and Mori, Kenichi "Pattern Identification Systems Operating by the Multiple Similarity Method", United States Patent 3,688,267, and Tokyo Shibaura Electric Co., Limited, August 29, 1972 From a old list of Polhemus patents Inner-product comparison (pixel map) OCR character recognition against standard reference template patterns, RMS average of match scores compared to threshold [IijimaT75] Iijima, Taizo and Mori, Kenichi "Pattern Identification System", United States Patent 3,906,446, assigned to Taizo Iijima and Tokyo Shibaura Electric Co., Limited, Tokyo, Japan. Cited in MaedaK87 OCR of characters (handwritten?) using sums of squares of values for similarity: improvement over patent 3,688,267. [IngallsD75a] Ingalls, Dan "Bit BLT", Xerox PARC Inter-Office Memorandum to Alto Users, November 19, 1975 First description of BIT BLT routines for Xerox Alto computer: Bit Block Transfer. Main operation is FillBitMap(...). includes Source code with date of April 20, 1976 [IrbyCH74a] Irby, Charles H. "Display techniques for interactive text manipulation", Proc. NCC '74, Chicago May 6-10, pp NLS text editing with mouse for text selection and keyboard: could show two (text) files at once, multiple users edit same text from separate terminals. 8:26:14 PM]

38 ( ) History of Pen and Gesture Computing [Ishizaki74] Ishizaki, Hiroyuki; Toba, Teruo; Umeda, Shozo "System for Reading Out the Coordinates of Information Displayed on a Matrix Type Display Device", United States Patent 3,832,693, August 27, 1974, Assigned to Fujitsu Light pen digitizer, subdividing display into blocks in the X and then in the Y direction. [Jans75] Jans, C.L. "An Investigation of U-V Discrimination", S.B. Thesis, Department of Electrical Engineering and Computer Science, May 1975, Massachusetts Institute of Technology, Cambridge, Massachusetts. U-V recognition: functional attributes Blesser's and Shillman's group at Massachusetts Institute of Technology [JohnsonRG72a] Johnson, Ralph G. and Fryberger, David "Touch Actuable Data Input Panel Assembly", United States Patent 3,673,327, June 27, 1972 Optical touchscreen designs. Light beams in X/Y grid over CRT display, finger interrupts and X and a Y beam. Internal reflections in transparent deformable layer (compressible silicone rubber), finger pressing on layer interrupts internal beams. Also sonic Rayleigh waves in surface of glass detected acoustically. [JohnsonS75] Johnson, S.C. "YACC: Yet Another Compiler Compiler", Computing Science Technical Report No 32, 1975, Bell Laboratories, Murray Hill NJ UNIX documentation on YACC; regular-expression pattern matching [JohnstonVS74] Johnston, V.S., and Chesney, G.L. "Electrophysiological Correlates of Meaning", Science, Vol 186, December 1974, pp Neuronal activity, evoked potentials in frontal lobe (but not visual cortex), matches human perception/recognition of recognizing ambiguous character shapes [Julesz75] Julesz, B. "Experiments in the Visual Perception of Texture", Scientific American, April 1975, pp "It is well known from neuro-physiological studies that the visual systems of cats and monkeys incorporate cluster detectors in several stages of hierarchically increasing complexity, beginning at the retina of the eye and extending to the highest levels of the cerebral cortex." Why adaptive recognition is not human perception/cognition: multiple levels Perception (low-level, immediate) vs cognitive (high-level, takes time) Mentions cognitive/perceptual problem of reading alphabets [KaplowR73a] Kaplow, Roy; Schneider, David; Smith, Franklin C. Jr.; and Stensrud, William R. "Computer Assistance for Writing Interactive Programs: TICS", ACM SIGCUE Computer Uses in Education Bulletin Vol 7 No 2, April 1973, pp TICS system for authoring tutorials for interactive education. Cites to Coursewriter. Examples include foreign-language instruction. Courses structured as web of nodes, connected by multiple possible branches. Compare with TutorText early 1960s. [KayA72] Kay, Alan C. "A Personal Computer for Children of All Ages", Xerox Palo Alto Research Center: Proceedings of the ACM National Conference, 1972 Early paper by Alan Kay on Dynabook, does not show handwriting input: touchscreen (strain gauges at corners) allows on-screen virtual keyboard display, some references to drawing. Early personal computer with ARPANet/Internet. [KnowltonKC75a] Knowlton, K.C. "Virtual Pushbuttons as a Means of Person-Machine Interaction", Proceedings of the IEEE Conference on Computer Graphics, Pattern Recognition, and Data Structure, May 1975, pp Cited in FoleyJD82 [KnowltonKC75b] Knowlton, K.C. "Interactive Input-Output Computer Terminal with Automatic Relabeling of Keyboard", United States Patent 3,879,722, assigned to Bell Laboratories, April 22, 1975 Re-label keyboard by projecting images on top. Compare with Kaplow. See Knowlton77 [Kolers75] Kolers, Paul A. and Perkins, David N. "Spatial and Ordinal Components of Form Perception and Literacy", Cognitive Psychology, Vol 7, 1975, pp Graphical context on rotated text perception with human readers: relation to dyslexia. [KorolevE71] Korolev, E. "On automatic recognition of context", Proceedings of the International Conference on Computer Linguistics, 1971 Cited in Suen79 [Krause73] Krause, P. and Bleichrodt, H. "Experiments on direct input and recognition of handwritten digits and handprinted letters with computers", Post Office Research Department, London, England, 9-16, March 1973 Cited in Blatt88: 97.1% character recognition rate [KuipersJ75] Kuipers, Jack "Object Tracking and Orientation Determination Means, System and Process", United States Patent 3,868,565, February 25, 1975 Polhemus patent on generating rotating fields to digitize position and angular position at same time (3-D tablet plus rotation, theta, phi, rho), six-dimensional electromagnetic digitizer (3,868,564 is a motor patent from the USSR) [KuklinskiT73] Kuklinski, T. "Goodness-Time Experiments", internal report, Cognitive Information Processing Group, Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, May 18, :26:14 PM]

39 ( ) History of Pen and Gesture Computing Cognitive experiments showing correlation between human rating of "goodness" of paired character shapes, and recognition time [KuklinskiT74] Kuklinski, T., draft manuscript "Plasticity Effects in the Perception of Handprinted Characters", Massachusetts Institute of Technology, Cambridge, Massachusetts, February 20, 1974 Graphical context affects perception of characters, not just semantic context [KuklinskiT75] Kuklinski, T., S.M. and E.E. Thesis "Plasticity Effects in the Perception of Hand-printed Characters", Massachusetts Institute of Technology, Cambridge, Massachusetts, 1975 Graphical context affects perception of characters, not just semantic context [Labov72] Labov, W. "The Boundaries of Words and Their Meanings", Conference on New Ways to Analyzing Variation in English, Washington, D.C., October 28, 1972 Variability/uncertainty a fundamental property of formal description Linguistics/perception/cognition as study of boundaries, not categories Justification of fundamental philosophy for our boundary-cognitive Analysis of functional attributes/features Feature analysis: not single property, but co-occurrence of large number of items [LarsonWA73a] Larson, Willis A. "Touch Sensitive Electronic Switching", United States Patent 3,737,670, June 5, 1973 Touch sensor, two electrodes (shaped to match finger tip): conduction by (high resistive) skin when electrodes touched. Alternative is to put elastomeric film above sensors, bottom layer conductive, finger presses film to the two electrodes. Not capacitive. [LeeT75] Lee, T.K-Y. "The Phenomenon of Line Addition in Character Recognition", S.B. Thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, May, 1975 Y-V discrimination: functional attributes Barry's and Shillman's group at Massachusetts Institute of Technology [Lemelson75] Lemelson, Jerome H. "Scanning System and Method", United States Patent 3,918,029, November 4, 1975 Optical hand-held scanner using a single sensing element, straight-edge or other guides for moving the stylus across a page: compare to optical digitizers [Lesk75] Lesk, M.E. "Lex - A Lexical Analyzer Generator", Computing Science Technical Report No 39, 1975, Bell Laboratories, Murray Hill NJ UNIX documentation on LEX: regular-expression pattern matching [Lesser75] Lesser, Victor, Fennel, Richard, Erman, Lee and Reddy, Raj "Organization of the Hearsay II Speech Understanding System", IEEE Trans. Acoustics, Speech and Signal Processing, Vol ASSP-23 No 1, pp 11-24, 1975 Cited in Purcell77 [LinWC74] Lin, W.C. and Scully, T.L. "Computer Identification of Constrained Hand-printed Characters with a High Recognition Rate", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-4, November 1974 Statis/OCR handwriting/handprinted character recognition, if first level classifier has no clear winner (20 weighted features), uses factors of two highest-scored candidates. 20-segment constrained handwriting style, Gaussian point-to-line distance to segments. Limited punctuation: period/dot does not fit onto segments. [Locke71] Locke, John L. and Locke, Virginia L. "Deaf Children's Phonetic, Visual, and Dactylic Coding in a Grapheme Recall Task", Journal of Experimental Psychology, 1971, Vol 89 No 1, pp Study of human character recognition for handwritten letter pairs, comparing the effects of auditorally similar letters to visually similar letters in deaf children. Refers to phonetic coding in memory. [MachineDesign72a] Machine Design "The Instant Draftsman", Machine Design magazine, April 6, 1972, pages Cited in Fitzgerald76 patent, example of sketch recognition / sketch processing [Malevard72] Malevard, Lucien C.; Marty, Pierre "Graphic Data Transcription system", United States Patent 3,632,874, January 4, 1972 Resistive sheet/film digitizer, using a single sheet and a conductive stylus and wire. Mentions integration with a display by applying the measured voltage directly to the deflection plates of a CRT. [Manis71] Manis, Melvin "Context Effects in Communication -- Determinants of Verbal Output and Referential Decoding", in "Adaptation-Level Theory", New York: Academic Press, M.H. Appley, editor, 1971, pp Theory for perceptual interpretation, subjective impression for context Kuklinski has margin notes on psycho-physical (perception), judgmental, and linguistic context Refers to other work on semantic (language) context Page relates to inadvertent bias in testing recognition results, since judgment affected by the rest of the characters the subject looks at (or writes) Page Kuklinski's margin notes on all the forms of context bias [MartinWA71a] Martin, William A. "Computer Input/Output of Mathematical Expressions", 2nd Symposium on Symbolic Algebraic Manipulation, A.C.M., March 23-25, 1971, Los Angeles, California, pp 78-ff. 8:26:14 PM]

40 ( ) History of Pen and Gesture Computing Cited in FoleyJD82: for user-interface for mathematical input, two-dimensional. Illustration shows use of character extents, spacing, size, position, for parsing two-dimensional hand-written mathematical formulae. [Meisel72] Meisel, W.S. "Computer-Oriented Approaches to Pattern Recognition", Academic Press, New York/San Francisco/London, 1972 (partial copy) Compares "abstract" issues of pattern recognition with heuristic/linguistic approach specific to problem area Feature selection by statistics "gives no insight into dependencies among features" Inter-set and intra-set feature distances an ill-defined concept "curse of dimensionality": how feature interact and overlap a hard problem Project features onto one dimension and you cannot tell how features inter-relate [MillerLK72] Miller, Leon K. "Letter Recognition: Effects of interitem similarity and report requirements", Perception and Psychophysics, Vol 11 No 3, 1972, pp Human recognition/perception: human readers (non-arabic, non-hebrew, non-chinese) perceive characters to the left of the fixation better than to the right [MillerSW71] Miller, S.W. "Display Requirements for Future Man-Machine Systems", IEEE Trans. Electronic Display, Vol ED-18 No 9, September 1971, pp 616-ff. CRT displays are too large, and resolution too low for many application (ca. 1971) [MitchellRJ74a] Mitchell, R.J. "Pressure Responsive Resistive Material", United States Patent 3,806,471, April 23, 1974 Pressure-sensitive conductive rubber: conductive particles in an elastomeric binder [Moayer75] Moayer, B. and Fu, K.S. "A syntactic approach to fingerprint pattern recognition", Pattern Recognition, Vol 7, 1975, pp 1-23 [MoriwakiM75a] Moriwaki, Masahiro, Sakaguchi, Mitsuhito, and Mita, Yoshinari "Hologram Graphic Data Tablet Apparatus Using a Vernier", United States Patent 3,906,465, September 16, 1975 Nominally optical digitizing tablet using hologram patterns printed in a glass surface. [MossD75a] Moss, David "Character Recording System", United States Patent 3,903,502, September 2, 1975 Character recognition for simplified character set using endpoints/termini only, which had to touch one of six reference points/zones in a hexagonal pattern [MottEM72a] Mott, E.M. "A Transparent Touch-Screen Device for Interactive Computer-Graphics Displays", Report RHEL/R 248, Rutherford High Energy Laboratory, 1972 Infrared-optical touchscreen [MuellerRE74a] Mueller, Robert E. "Direct Television Drawing and Image Manipulating System", United States Patent 3,846,826, November 5, 1974 Optical drawing input (analog digitizer) using optical prisms that can respond to any touching object: rubber stamps, fingers, brushes: light affects flying-spot scanner for a "paint-pulse" input. Also shows piano-type keyboard for selecting visual effects. Later cited by Greene for multi-finger/object (multi-touch) drawing system using same principle. [Nakata72] Nakata, K., Nakano, Y., and Uchikura, Y. "Recognition of Chinese Characters", Proc. of Conference on Machine Perception of Patterns and Pictures, pp (1972) OCR by projecting in X and in Y direction for histogram [NarasimhanR71a] Narasimhan, R. and Reddy, V.S.N. "A Syntax-Aided Recognition Scheme for Handprinted English Letters", Pattern Recognition 3, pp , 1971 [NaylorWE71a] Naylor, W.C. "Some studies in the interactive design of character recognition systems", IEEE Trans. Computing, Vol 20, September 1971, pp Cited in Blatt88: 80.4% character recognition rate [NegroponteN72a] Negroponte, Nicholas; Groisser, Leon B.; and Taggart, James "HUNCH: An Experiment in Sketch Recognition", "Evironmental Design: Research and Practice", Proc. EDRA 3 / AR 8 Conf., Univ. California at Los Angeles, Sketch recognition as early problem in artificial intelligence. Includes recognition of gestures, adaptation to user, pen pressure on Sylvania tablet. Uses report rate from tablet as clock to determine dwell. Pressing harder makes lines darker (user wants to show that line is correct), Z proximity sensing. Transparent touchscreen, first pressure-sensing pen/stylus? System asks for user sketching samples (e.g. slow and fast squares) for initial training to user. Squiggles/rub-out gesture. Adaptive thresholds on pressure/force. [NegroponteN73a] Negroponte, N. "Recent Advances in Sketch Recognition", NCC 1973, AFIPS Press, pp Cited in FoleyJD82 Sketch recognition as a form of data compression: primary example is recognizing straight lines, simple curves. Constraints (two lines parallel, same length, etc.) [NegroponteN75a] Negroponte, Nicholas "Sketching: A Computational Paradigm for Personalized Searching", Working Paper, Architecture Machine Group, Department of Architecture, Massachusetts Institute of Technology, Cambridge, Massachusetts, Also Jnl of Architectural Education, vol 29 no 2, 1975 pp (partial copy) Indexing by handwritten electronic-ink user-interface using sketch recognition / sketch matching [NelsonTH74a] Nelson, Theodor H. "Dream Machines: New Freedoms through Computer Screens -- A Minority 8:26:14 PM]

41 ( ) History of Pen and Gesture Computing Report - The New Republic of PLATO / Computer Lib: You Can and Must Understand Computer's Now", Nelson, Theodor Computer Lib: You Can and Must Understand Computers Now; Dream Machines: New Freedoms Through Computer Screens- A Minority Report. Self-published. ISBN Speculative, comic-book style publication on future computer technology: long section on PLATO touchscreen terminals for education. PDP-11 GRASS system with Vector General tablet. Text editing with light pen (Parallel Textface application) showing multiple windows with sections of same document, for editing hyperlinks. Cites to Ledeen character recognizer. [NemirovskySM71a] Nemirovsky, S.M., Sternberg, J., and Lieberman, G. "Signature Identification by Means of Pressure Patterns", United States Patent 3,618,019, November 2, 1971, assigned to Conversational Systems Corporation, New York, New York. Signature verification, mis-called signature identification, using pressure pattern without position. Pressure transducer is strain gauge or piezoelectric sensor in stylus/pen, etc, alternative the tablet/platen is a force/pressure transducer. [Neuhoff75] Neuhoff, D.L. "The Viterbi algorithm as an aid in text recognition", IEEE Trans. Information Theory, Vol IT-21, 1975, pp Cited in Sinha88 [NewmanWM73a] Newman, W.M. and Sproull, R.F. "The Ledeen Character Recognizer", in Principles of Interactive Computer Graphics, McGraw-Hill, New York, 1973, pp , Appendix VIII The handwriting/gesture symbol recognition system used by Applicon CAD/CAM system: everyone used to copy it. Trainable. Refers to training to recognize a full alphabetic character set, but with some neography required. [NewmanWM73b] Newman, W.M. and Sproull, R.F. "Principles of Interactive Computer Graphics, second edition", ISBN , McGraw-Hill, New York, 1973, 1979 (hardcopy book) Chapter eleven: cannot use mouse for tracing, compared with digitizing tablet. Mentions RAND tablet (electromagnetic), voltage-gradient (resistive tablet), Sylvania tablet (electrostatic/capacitive), acoustic tablet (SAC: see Science Accessories Corporation) [NewmanWM73c] Newman, W.M. and Sproull, R.F. "Tablets: Principles of Interactive Computer Graphics, second edition", ISBN , McGraw-Hill, New York, 1973, 1979 Scan of book excerpt: RAND tablet, etc. [Nishida72] Nishida, Nobuo and Sakaguchi, Mitsuhito "Hologram Graphic Data Tablet", United States Patent 3,658,402, April 25, 1972 Digitizing tablet, nominally optical: holograms embedded at different positions on the tablet [NollAM75a] Noll, A. Michael "Tactile Man-Machine Communication System", United States Patent 3,919,691, November 11, 1975 Electromechanical 3-D rod (3DOF) with force feedback to simulate surfaces haptically. [OkaM74x] Oka, Masatomo and Yasuhara, Makoto "Signature Verification Experiment Based on Nonlinear Time Alignment: A Feasibility Study", Research Manuscript, 1974 (date approximate), Nippon Electric Company, Tokyo Japan From Kuklinski's old files Dynamic time-warping / non-linear time alignment via dynamic programming for signature verification: published later in 77/78 [OrganickEI72a] Organick, Elliot I. "The Multics System: An Examination of its Structure", The MIT Press, 1972 Architectural overview and design of the MULTICS system. Paging using associative memory tables. Runtime (delayed) linking and loading. Ring/bracket protection of entry points. Interprocess communication. Hardware support for access rights (access control), ACL / access control lists on files and objects. Access and mode control on I/O devices. Not much on networking. [PapenmeierFH75a] Papenmeier, F.H. "BRAILLEX - Das erste Gerät mit einer elektronischen Braillezeiler", Papenmeier.de, fetched 2015 Historical notes on Papenmeier BRAILLEX, early/first tactile Braille display. [Pavlidis74] Pavlidis, Theo and Horowitz, S.L. "Segmentation of Plane Curves", IEEE Trans. Computers, Vol C-23 No 8, 1974, pp Best piece-wise fit of line segments to a polygon: split and merge segments to drive error below a threshold Piece-wise line segment fit helps on feature extraction, data compaction, and noise filtering [Pavlidis75] Pavlidis, Theo, and Ali, F. "Computer Recognition of Hand-written Numerals by Polygonal Approximations", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-6, pp , 1975 Chain codes - cite this one so Pavlidis gets his licks in [PerlmanR74a] Perlman, Radia "TORTIS (Toddler's Own Recursive turtle Interpreter System)", Report LOGO-9, M.I.T. Artificial Intelligence Lab, March 1974 Logo graphical turtle, with control interface with physical buttons for toddlers: buttons labeled with command and numbers, with a memory, so see whether toddlers (ages 4-6) would learn to program. Cited in Minsky84 8:26:14 PM]

42 ( ) History of Pen and Gesture Computing [PeronneauG72a] Peronneau, Georges and Poizat, Henri "Position-Indicating System", United States Patent 3,657,475, April 18, 1972 Touch-panel digitizer using force gauges/sensors at three or four corners of a plate: compute location of touch from corner forces. Plate can be transparent, and sensors are only at the corners. [PobgeePJ71] Pobgee, P.J. and Parks, J.R. "Applications of a Low Cost Graphical Input Tablet", IFIP 1971, North- Holland, Amsterdam, pp??, (TA-4-169) Cited in FoleyJD72 [PobgeePJ75a] Pobgee "Graphical Input Apparatus for Electrical Apparatus", United States Patent 3,885,097, May 20, 1975 Deformable resistive sheet digitizer with insulating mesh between so that resting hand does not cause sheets to make contact. Palm rejection. [PowersVM72a] Powers, V.Michael "Handwritten Character Recognition from Pen Direction", Naval Postgraduate School, Monetery CA April 21, 1972 Describes character recognition based solely on chain codes / stroke direction / arc segments [PowersVM73a] Powers, V.Michael "Pen Direction Sequences in Character Recognition", Pattern Recognition, Vol 5. pp , March 1973 Describes character recognition based solely on chain codes / stroke direction [Quarmby71] Quarmby, David J. and Rastall, John "Experiments on Handwritten Numeral Classification", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-1 No 4, October 1971, pp Edge-defining algorithms for OCR of handprinting/handwriting, maximum likelihood statistical classifier [Ramer72] Ramer, U. "An iterative procedure for the polygonal approximation of plane curves", Computer Graphics Image Processing, Vol 1, November, 1972, pp Cited in Burr81 [RebaneGJ74a] Rebane, George J. and Grossman, Norman R. "Graphic Input Device", United States Patent 3,875,022, December 24, 1974 Rear-projection display on translucent acoustic time-delay touchscreen digitizer, receivers for sounds of tap/contact of stylus with the digitizer screen. Sound generator in stylus tip. [RichardsonS72a] Richardson, S.; Chan, H.L.; Lee, A.; and Teo, S.T. "The Mueller-Lyer Illusion: A Cross-Cultural Study in Singapore", Ergonomics, Vol 15 no 3, 1972, pp Study showing that culture affects optical perception: Singapore Chinese more subject to Mueller-Lyer linelength illusion than Singapore British [RisemanEM71a] Riseman, E.M. and Ehrich, Roger W. "Experiments in the contextual recognition of cursive script", IEEE Trans. Computers, Vol C-24, 1975, pp Cited in Goshtasby88 for binary letter pairs (digrams) for spelling context, following by spelling dictionary look-up for what's left, compare with Viterbi algorithm or with trigrams [RisemanEM71b] Riseman, E.M. and Ehrich, Roger W. "Contextual word recognition using binary digrams", IEEE Trans. Computers, Vol C-20, April 1971, pp Cited in DosterW77 on a spelling dictionary: important point of diagram is only zero/non-zero value [RisemanEM71c] Riseman, Edward M. "Logical Networks for Feature Extraction", IEEE Trans. on Sytems, Man, and Cybernetics Vol CMC-1 no 1, January 1971, pp. 43-ff Automatic determination of useful features for OCR character recognition (on thinned images) by mathematical determination of clusters matching desired outcomes. Notes that the test data set was unrealistically small training set. [RisemanEM73a] Riseman, E.M. and Ehrich, Roger W. "Contextual word recognition using binary digrams", IEEE Trans. Computers, Vol C-20, April 1971, pp Preliminary abstract of Fisher75 [RisemanEM73b] Riseman, E.M., Hanson, A.R. and Fisher, E.G. "Feature Selection Using Thresholded Measures", Proceedings of 1973 International Conference on Cybernetics and Society, Boston, Massachusetts, November 5-7, 1973 [RisemanEM74] Riseman, E.M., and Handson, A.R. "A contextual postprocessing system for error correction using binary N-grams", IEEE Trans. Computers, Vol C-23, 1974, pp (abstract only) Cited in DosterW77 on a spelling dictionary. File also contains list of numerous reference *to* Riseman for context information in handwriting and speech recognition. [RodgersJL73a] Rodgers, James L. "Apparatus for Converting the Position of a Manually Operated Instrument into an Electrical Signal", United States Patent 3,767,858, October 23, 1973 Electrostatic grid digitizer, same signal at same frequency fed on one side and other side but at different phase, stylus detects phase angle across tablet: resistor ladder to apply phase shift on each conductor. Vertical and horizontal elements of grid/matrix operate at different frequencies simultaneously, so X and Y determined simultaneously. [RodgersJL75a] Rodgers, James L. and Tyrrell, Josph L. "Electrical Writing Pen and Sensor", United States Patent 8:26:14 PM]

43 ( ) History of Pen and Gesture Computing 3,886,311, May 27, 1975 Stylus design of capacitive/electrostatic grid digitizer, tablet is the transmitter. Cord on stylus. [RosenfeldA71a] Rosenfeld, A. and Thurston, M. "Edge and curve detection for digital scene analysis", IEEE Trans. Computers, Vol C-20, May 1971, pp Cited in Teh89 [RosenfeldA72a] Rosenfeld, A., Thurston, M., and Lee, Y.H. "Edge and curve detection: further experiments", IEEE Trans. Computers, Vol C-21, July 1972, pp Cited in Teh89 [RosenfeldA73a] Rosenfeld, A. and Johnston, E. "Angle detection on digital curves", IEEE Trans. Computers, Vol C-22, September 1973, pp Cited in Teh89 [RosenfeldA75a] Rosenfeld, A. and Weszka, J.S. "An improved method of angle detection on digital curves", IEEE Trans. Computers, Vol C-24, September 1975, pp Cited in Teh89 [RosenfieldJM73] Rosenfield, John M., Cranston, Fumiko E., and Cranston, Edwin A. "Japanese Language Calligraphy", in The Courtly Tradition in Japanese Art and Literature, Fogg Art Museum, Harvard University, 1973, pp Background history of Chinese/Japanese Kanji, Katakana, hiragana: educated Japanese cannot recognize Kana scripts [SaltzerJH74a] Saltzer, Jerome H. "Protection and the Control of Information Sharing in Multics", CACM Vol 17 No 7 July 1974, pp Overview of protection mechanisms in Multics and file system: ACL access control lists, list of user/project/compartment (extendible) and access rights, associated with object. (Capability system would associate with user.) Description of "*" wildcard/don't-care syntax. [SaltzerJH74b] Saltzer, Jerome H. "The Protection of Information in Computer Systems", Proc. Fourth ACM Symposium on Operating System Principles, October 1973 (alternate version of CACM 1974 paper) Overview of protection mechanisms in secure computer systems: ACLs access control lists embedded in indirect objects -- separate from object for which permission needed, separate access controller granting access, looks for match. Mentions encipherment/encryption (notes that security classifications from military restrict publication of ideas): authentication by handwriting signature verification or rhythm of keyboard typing (biometrics), physical tokens (unduplicateable identity card/token). Multi-node networked systems. [SaltzerJH75a] Saltzer, Jerome H. and Schroeder, Michael D. "The Protection of Information in Computer Systems", Proc. IEEE, Vol 63 no 9, September 1975, pp (alternate version of CACM 1974 paper) IEEE journal version of 1974 ACM paper. Overview of protection mechanisms in secure computer systems: ACLs access control lists embedded in indirect objects -- separate from object for which permission needed, separate access controller granting access, looks for match. Mentions encipherment/encryption (notes that security classifications from military restrict publication of ideas): authentication by fingerprint reader, handwriting signature verification or rhythm of keyboard typing (a.k.a. keystroke dynamics) (biometrics), physical tokens (unduplicateable identity card/token). Multi-node networked systems. Quote: "Work in progress is not well represented by published literature." [Sayre73] Sayre, Kenneth M. "Machine Recognition of Handwritten Words: A Project Report", Pattern Recognition 5, pp , % handwriting recognition on 84 cursive script words (OCR) Does cursive script recognition as part of handwriting recognition (OCR) Statistical recognition (digrams) can make recognition worse: "fo" more frequent than "fa" in English, so ambiguous "far" would always be recognized wrong as "for" 90% per character gives 81% per pair, 72% per triplet, etc.: fix with context Used handwriting data samples from Bell Telephone Labs (OCR) Cites very early work by Harmon, internal Bell Telephone Labs reports [Schlang74] Schlang, Arthur "Electro-Optical Handwritten Character Reader", United States Patent 3,784,982, assigned to Isotec, Incorporated, Woodbury, New York, January 8, 1974 Cited in Tappert88b: on segmentation combined with character recognition OCR scanner for handwritten characters: hardware to detect inflection points, endpoints, sub-stroke (horizontal and vertical segments) of strokes Describes two-level recognizer with (generic) first-level and detail-level feature analysis [SearsFW74a] Sears, Francis W.; Zemansky, Mark W.; and Young, Hugh D. "College Physics, Fourth Edition", Addison-Wesley, 1974 (hardcopy book) Standard physics reference and textbook [Shepard72] Shepard, Roger N. "Psychological Representation of Speech Sounds", in "Human Communication: A Unified View", Edward E. David, Jr. and Peter B. Denes, editors, McGraw-Hill Book Company, 1972, pp 67-ff. Confusion data matrix on phoneme recognition/discrimination in speech recognition: degree of similarity, confuseability, association, psychological "proximity": how to find out a distance metric for pattern recognition. proximity analysis, signal-to-noise ratio in pattern space, multi-dimensional scaling 8:26:14 PM]

44 ( ) History of Pen and Gesture Computing [ShillmanR74a] Shillman, R., Kuklinski, T.T., and Blesser, B.A. "Experimental Methodologies for Character Recognition Based on Phenomenological Attributes", Proceedings of the 2nd International Joint Conference on Pattern Recognition, Copenhagen, Denmark, August 13-15, 1974, pp functional attributes reliable (e.g. tail) on block character recognition: graphical context biases the attribute equally for all characters. Did not include context where only certain characters with the attribute were shifted? [ShillmanR74b] Shillman, R., Cox, C., Kuklinski, T., Ventura, J., Eden, M., and Blesser, B. "A Bibliography in Character Recognition: Techniques for Describing Characters", Visible Language, Vol VIII No 2, Spring 1974, pp General references on character recognition, especially cognitive psychology aspects Includes references on insight through study of character formation / generative variability Includes references on psychological descriptions of characters [ShillmanR74c] Shillman, R.J. "Seeing is Deceiving: The Perceived Shortening of Thick Lines", manuscript in preparation, Cognitive Information Processing Group, Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, 1974 Brief user study of apparent line shortening for wider/narrow characters: graphical context on character recognition. [ShillmanRJ74] Shillman, R. "Character Recognition Based on Phenomenological Attributes: Theory and Methods", PhD Thesis, Massachusetts Institute of Technology, Department of Electrical Engineering, 1974 Paper copy only. Lists features that are not perceptually significant Shillman's Thesis: also published as BlesserB73 [ShillmanRJ75a] Shillman, Robert J. "Hand Printed Digit Reading under Conditions of Limited Control", Draft manuscript, Massachusetts Institute of Technology, 1975 OCR for discrete boxed handwriting character recognition of digits with cost and accuracy of keypunch typing transcription manually [ShinarD73a] Shinar, David and Owen, Dean H. "Effects of form rotation on the speed of classification: The development of shape constancy", Perception and Psychophysics, Vol 14 No 1, 1973, pp Rotationally invariant (at 90 degree increments) recognition of simple geometric shapes, apparently by learning shapes in each orientation: not specific to character recognition. [SimonJC72a] Simon, J.C., Checroun, A. and Roch, C. "A Method of Comparing Two Patterns Independent of Possible Transformations and Small Distortions", Pattern Recognition, Vol 4, 1972, pp Cited in Baird84: for template matching in feature space using minimum distance? [SiyP74] Siy, Pep, and Chen, C.S "Fuzzy Logic for Handwritten Numeral Character Recognition", IEEE Trans. on Systems, Man, and Cybernetics, November 1974, pp Trainable system using subsets of curved line segments as the 'branch feature set' [SmithEE74] Smith, E.E. and Spoehr, K.T. "The Perception of Printed English: a Theoretical Perspective", in "Human Information Processing", B.H. Kantowitz, editor, Lawrence Hillbaum Associates, Hillsdale NJ 1974, pp Template matching vs feature analysis is a continuum, not disjoint Cites several papers on low-level feature extraction in human perception Argument against dynamic matching: "theoretical and empirical reasons for feature extraction" Cites different recognition mechanism for handwritten and printed characters Cites no real consensus on actual features humans use in recognizing letters (which is why functional attribute tests tell you what features) [SpanjersbergAA74a] Spanjersberg, A.A. "Combinations of different systems for the recognition of Handwritten digits", Proceedings of the Second International Joint Conference on Pattern Recognition, August, 1974, pp Cited in Blatt88: 97.8% character recognition rate [SpillaneRJ75a] Spillane, R.J. "Keyboard Apparatus for Personal Identification", IBM Technical Disclosure Bulletin 04-75, p. 3346, April 1, 1975 Biometric authentication of users based on typing pattern while entering standard entry code (password/pin): time pattern and key pressure (force) characteristics. See also Saltzer Compares to telegraph key signal ("Fist"). [SteinmannH74a] Steinmann, Helmut "Capacitive Detector Device", United States Patent 3,826,979 Capacitive sensor, used to detect presence of water (or similar fluid) on the other side of a car windshield, and turn on the windshield wipers. Distinguishes between different materials, e.g. does not respond to debris on the windshield. [StruvenWC73a] Struven, W.C. "Experience with Touch Panel Control at SLAC", SLAC technical report SLAC-PUB (A), March x512 cross-wire (capacitive?) touch panel digitizer on CRT display, menu buttons arranged 10 x 13. Refers to "panel" as term for hierarchy of menus 8:26:14 PM]

45 ( ) History of Pen and Gesture Computing [Suen73a] Suen, Ching Y. "Factors Affecting the Recognition of Handprinted Characters", Proc. Intl. Conf. on Cybernetics and Society, November 1973, pp OCR recognition of handwriting affected by scanner, lighting, pencil type. Handwritten data collection done in the field, where computer not available for data entry. [Suen74a] Suen, Ching Y. "Human Factors in Character Recognition", Proc. Conf. and Symp. on Systems, Man and Cybernetics, Octb 1974, pp Study of legibility of 30 different handwriting styles taught in North America. Block printing more legible than cursive (in general). [Suen75] Suen, C.Y. "Handwriting Education - A Bibliography of Contemporary Publications", Visible Language, Vol IX No 2, Spring 1975, pp Refers to unpublished treatise on handwriting style variations in North America [SutherlandIE72a] Sutherland, Ivan E. "Display Windowing by Clipping", United States Patent 3,639,736, February 1, 1972 Clipping of drawn lines to a viewport window by binary division of the lines to edges of window. [SutherlandIE74a] Sutherland, I.E. "Three-dimensional Data Input by Tablet", Proceedings of the IEEE, Vol 62 No 4, April 1974, pp (abstract only) multiple-pen user input on large-area tablet digitizer, three dimensional 3D input by providing input for multiple perspective views simultaneously. Compare with Buxton multi-hand/two-hand input, multi-touch. [SutherlandNS73] Sutherland, N.S. "Object Recognition", Chapter 8 of "Handbook of Perception" Carterette and Firedman, ed., Academic Press, New York and London, 1983 General essay on human pattern recognition. Visual recognition of objects: segmentation of images, grouping of features, work for varying rotation and orientation. Cites multiple studies that recognition is by syllable rather than by phoneme. [Szanser72] Szanser, A.J. "Elastic matching in automatic pattern recognition", in Machine Perception of Patterns and Pictures, Teddington, April 12-14, 1972, pp Cited in DosterW77 on spelling dictionary [SzanserAJ73a] Szanser, A.J. "Bracketing techniques in elastic matching", Computer Journal, Vol 16 No 2, pp , 1973 Elastic matching applied to general strings of characters/codes: sped up by assuming initial part of pattern to be matched is not garbled, bracketing occurences of that sequence in input string. [TaggartJR73a] Taggart, James Richard "Reading a Sketch by HUNCH", Master Dissertation, EE., M.I.T. May 1973 Data-reduction in sketch recognition. Corners are important feature, detected better when taking into account drawing speed -- jitter when drawing at low speed badly affected line-straightening. Proposal to use pressure: slowly drawn or heavy pressure means user input is more deliberate. Sylvania data tablet, transparent tablet/touchscreen, two distance levels for proximity sensing. Storage-tube display (hard to erase, compared to drawing with physical ink). Replay sketching data at same timing as input -- compare with interactive system such as Wang Freestyle? Proposal for graphical compiler to process (recognize) sketches: compare to 2D mathematical input? [TarnopolskyI74a] Tarnopolsky, Igor Rafailovich and Akimtsev, Gennady Vasilievich "Graphic answer input device for a teaching machine", United States Patent 3,793,469, February 19, 1974 diode-based DC touch panel / tablet for teaching machine, operates as cross-bar switch, can sense two distinct contact (multi-touch). Compare with Kaplow switching matrix for touchscreen? [TeraiH72a] Terai, Hidekazu and Nakata, Kazuo "On-line, Real-time Recognition of Handwriting Chinese (Kanji) Characters and Japanese (Katakana} Syllabary", Central Research Laboratory of Hitachi Ltd, 1972, Monograph, originally published in Japanese for Inst. of Electronics and Comm. Eng. of Japan, June 15, 1972 Recognition of Japanese/Chinese character using chain codes (direction codes) for first level classification of fundamental strokes [TeraiH75a] Terai, Hidekazu and Nakata, Kazuo "Text Editing System using On-Line Real-Time Handwriting Character Recognition", Central Research Laboratory of Hitachi Ltd, Sept, 1975, Monograph, originally published as full paper for Journal of Information Processing Society of Japan, Vol 15 No 6, June 1974 proof-reading symbol recognition, writing directly over the strokes of a character on a tablet character recognition input in a special area of the tablet Cited in Sklarew99 prior art [Thompson75] Thompson, K. and Ritchie, D.M. "UNIX Programmer's Manual -- Sixth Edition", Bell Laboratories, Murray Hill, New Jersey, 1975 General UNIX reference [Tou72] Tou, Y.T. and Gonzales, G.C. "Recognition of Handwritten Characters by Topological Feature Extraction and Multilevel Categorization", IEEE Trans. Computers, Vol 21, pp , July 1972 Cited in Goodale83 [TownsendJT71a] Townsend, J.T. "Theoretical analysis of an alphabetic confusion matrix", Perception and Psychophysics, Vol 9, 1971, pp :26:14 PM]

46 ( ) History of Pen and Gesture Computing Mathematical treatment of using confusion matrix with three different recognition algorithms: choice model, and finite sensory state models. Suggest that better character recognition will need a model of specific distinguishing features: compare to Pencept? [Turner72] Turner, John A. "Electrical Probe-position Responsive Apparatus and Method", United States Patent 3,699,439, October 17, 1972, assigned to Automatic Radio Manufacturing, Inc. Melrose MA Resistive-film digitizer, ratiometric measurement, mounted on front of CRT display [UllmanJR74] Ullman, J.R. "A use of continuity in character recognition", IEEE Trans. Systems, Man, and Cybernetics, 1974, pp Cited in Goshtasby88 using edge directions for recognizing hand-printed character if the continuity information between characters is utilized (stroke hooks and continuation marks?) [UncapherKW71] Uncapher, K.W. "The RAND Video Graphic System - An Approach to a General User-Computer Graphic Communication System", RAND Corporation Research Report R-0753-ARPA, April Presented to AGARD Avionics Panel Technical Symposium on Data Handling Devices, Istanbul, Turkey Jun 1-4, 1970 Copy on file has best image of RAND tablet and display hardware: shows photographic image and electronic text/ink displayed on same surface Multiple users on one graphics system: 32 consoles, each with up to 8 different input devices. Mentions development of all-digital video. Compare with later Schumer patent. [VanDam71] Van Dam, Andries and Rice, David E. "On-line Text Editing; A Survey", Computing Surveys, Vol 3 No 3, September 1971, pp 93-ff. Describes a tablet-based text editor from Carnegie-Mellon University (Coleman69), using symbols/gestures on a RAND tablet: decision tree to recognize symbols. Ease of correction makes requirement for perfect recognition be less. Cost of computers made handwriting recognition not cost-effective in [VolpeJW75a] Volpe, John W. "Capacitance Matrix Keyboard", United States Patent 3,921,166, November 18, 1975 Capacitive matrix keyboard, similar to capacitive touchscreen. Physical keys move bridging conductor near capacitive elements. [WaltzDL72a] Waltz, David L. "Generating Semantic Descriptions from Drawings of Scenes with Shadows", Ph.D. Thesis, Massachusetts Institute of Technology, Department of Electrical Engineering, 1972 Analysis of line-drawing images using combinatorial algorithms: compare permutation-free search with chain codes in handwriting recognition and structural grammars in sketch recognition [WardJR74a] Ward, Jean Renard "Computer-driven Tutorials on German word-order Using The TICS Computer Sub-System", SB Thesis, Course VI, M.I.T, May, 1974 Tutorial system for German grammar doing automatic grammatical analysis according to the two-prong word-order grammar of Lohnes and Strothmanm. [WeaverAC74a] Weaver, Alfred C. "On-Line Character Recognition", Department of Computer Science, University of Illinois at Urbana-Champaign, Report No UIUCDCS-R , August, 1974 Thesis (?) on proposed low-cost handwriting recognition product? Voltage-gradient (resistive sheet) tablet. Lists stroke directions for on-line characters [WeinbergGM71] Weinberg, Gerald M. "The Psychology of Computer Programming", Littleton Educational Publishing, 1971 (hardcopy book) Essays on the role of personality and real-world-practice versus methodology-theory in software development. Anniversary edition 25 years later compared state of progress with that of 1971 publication. [WeisgerberRA75a] Weisgerber, Robert A. "Evalaution of an Ink Print Reading Aid for the Blind: The Sterotoner. Final Report", Veterans Administration Report AIR /75-FR, December 1975 Aid for the blind: hand-held wand with optical sensor, device generates different pitch tones to indicate sensor reading. Slit camera (10 sensors vertical) user scans across printed material, generates tones for each sensor. Compare to Optacon. [WhetstoneA71a] Whetsone, Albert; Fine, Samuel; Bank, William; and Trumbull, Stanely Phillips "Spark Pen", United States Patent 3,626,483, December 7, 1971 SAC Science Accessories Corporation acoustic stylus for sonic digitizer: generated shock energy sound wave by spark. [WhetstoneA75a] Whetsone, A. and Domyan, S. "Recognize hand-printed characters with a simple algorithm", Electronic Design, February 1, 1975 Summagraphics original design of tablet for character recognition, zone based, numerics of single-strokes and double-strokes [WhetstoneA75b] Whetsone, Albert L. and Brenner, Alfred E. "Position Determination Devices", United States Patent 3,904,821, September 9, 1975 Summagraphics magnetostrictive tablet, using sheet of magnetostrictive material [Widrow73] Widrow, B. "The rubber mask technique, Parts I and II", Pattern Recognition, Volume 5, 1973, pp :26:14 PM]

47 ( ) History of Pen and Gesture Computing Cited in Burr81 pattern recognition using a template (e.g. shape/classifier of chromosomes in images) by allowing non-exactly-matching signatures: elastic matching on handwriting (?) recognition. [WilliamsTG71] Williams, Thomas G. "An On-Line System for Hand-Printed Input: Final Report for Phase IV", NASA Contractor Report CR , Washington, D.C., August 1971, prepared by System Development Corporation User writes in two dimensions, compiler translates to linear form (shown at top of display) an evaluates. Characters are translated into recognized characters in two-dimensional form for editing in real time. Uses bounding rectangles in parsing. Gesture recognition described, handwriting recognition is in a separate publication. "Scrub mode" is a scratch-out gesture. Arrow gesture indicates move/edit. Horizontal line gesture for "open space/move" editing operation. Double horizontal line for close up. Dot (tap) plus second line to move a character. Two perpendicular stroke gesture for select/move group. [WilliamsTG72a] Williams, Thomas G. "An On-Line two-dimensional computation system", Proc AFIPS '72, proc. of FJCC, pp The Assistant Mathematician (TAM), two-dimensional hand-printed mathematical input for computer programming: visual programming language VPL. Transparent/translucent digitizer tablet with rearprojected CRT display. Mathematic expression is displayed as recognized by system for confirmation by user in semantic context: semantic ambiguity of mathematical language (e.g. "i" as SQRT(-1), variable, or number). Scrubbing (rubout) gesture, editing of electronic ink. [WinstonPH72a] Winston, P.H. "The Massachusetts Institute of Technology Robot", in "Machine Intelligence 7", edited by B. Meltzer and D. Michie, pp , Edinburgh University Press, Edinburgh, Scotland, 1982 [WipkeW71a] Wipke, W. and Whetstone, A. "Graphic digitizing in 3-D", Computer Graphics (ACM Newsletter) Vol 5 No 4, 1971 Cited by ClarkJH76 as 3D digitizer [WolfbergMS72a] Wolfbeerg, Michael S. "AMBIT/L Programming System User's Guide", ARPA report 1228, CA , January 17, 1972 User's guide for AMBIT/L visual programming language / digrammatic language for list processing. Refers to Drawing of AMBIT/L Programs and Their Encodement. See AMBIT/G for lightpen/tablet user interface. [Woods73] Woods, Donald R. and Lyon, James M. "The INTERCAL Programming Language Reference Manual", privately published technical report, 1973 INTERCAL: The world's most bizarre, yet clearly defined, programming language (Geek humor). Compare with spoof on Basic at Data General, 1974? [Xebec74] Xebec Systems, Incorporated "Pattern recognition allows use of handwritten characters in data-entry system", IEEE Spectrum, new product section, Alphabec-70 handwriting recognizer, 566 Xavier Avenue, Sunnyvale, California 94086, October 1974, p 98 Product announcement on SRI/Xebec handwriting character recognition using accelerometer pen stylus. See Hewitt Crane [YamamotoS74] Yamamoto, Shinji and Nakata, Kazuo "Pattern Recognizing System", United States Patent 3,829,831, August 13, 1974, assigned to Hitachi Limited. Cited in Wang,PSP85 32x32 and 8x8 pixel bit-map character recognizer for OCR of Chinese [YasuharaM75a] Yasuhara, M. "Experimental Studies of Handwriting Process", Report of the Research Laboratory of Communication Science, University of Electro-Communications, Japan, Vol 25-2, (Science and Technology section), pp , March 1975 Mathematical model for pen dynamics in script, but not print or stroke connection [YhapEF75] Yhap, E.F. "Keyboard Method for Composing Chinese Characters", IBM Journal of Research and Development, Vol 19 No 1, May 1975, pp key special keyboard for input of Chinese Kanji characters, keys used to input shape of the character. [ZadehLA73a] Zadeh, L.A. "Outline of a new approach to analysis of complex systems and decision processes", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-3, pp 24-44, 1973 Biswas81 cites on fuzzy sets for recognition. Focus is on linguistic variables: e.g. height is a variable with possible values short, tall, very tall, extremely tall, medium, etc. A calculus of fuzzy sets instead of a precise calculus of numbers. [ZadehLA75a] Zadeh, L.A., Fu, K.S., Tanaka, K., and Shimura, M. "Fuzzy Sets and Their Applications to Cognitive and Decision Processes", London: Academic Press, 1975 Biswas81 cites on fuzzy sets for recognition. Compendium / graduate text of papers on fuzzy sets and fuzzy logic. Specific applications include handwriting recognition, fuzzy grammars and pattern recognition: compare with chain codes and parsing rules for sketch recognition. [ZadehLA75b] Zadeh, L.A. "The concept of a linguistic variable and its application to approximate reasoning -- I", Information Science, Vol 8 pp , 1975 Tanaka82 on fuzzy sets. Linguistic variable is variable who value is (overlapping/qualitative) set membership: e.g. age: young, not young, old, very old vs. 21, 22, 23. Applications to pattern recognition. 8:26:14 PM]

48 ( ) History of Pen and Gesture Computing [ZahnCT74a] Zahn, Charles T. Jr. "An Algorithm for Noisy Template Matching", Proceedings of IFIP 74, 1974, pp Post-processing of a distribution of features points in 2D Bayesian space, attempt to cover "noise" from translation, rotation, and noise displacement of the individual points. Cited in Baird84. Only tested with small number of artificial samples. [ZimmermannR72] Zimmermann, R. "Dynamische Beschreibung von Zeichenschreibweise und ihre Ausnutzung zur Erkennung von Handschriftzeichen", Nachrichtentechnische Zeitschrift, Vol 20 No 12, December 1972, pp Dynamic description of character drawing and its use for on-line recognition of handwritten characters (in German) zone and chain-code recognition for handwriting/hand-printing of symbols. Based on Doctoral thesis, T.U. Aachen Describes use of direction information in recognition, light pen on display. biblio70 (Prev page) Main Page (Next page) biblio80 A bargain is something you don't really need at a price you really can't resist. Manchmal braucht man für einen groben Klotz eben ein grobes Beil. 8:26:14 PM]

49 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1976 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:23 EDT Mirrors of this page can be found at: (Generally most up-to-date) biblio75 (Prev page) Main Page (Next page) biblio83 [Agui79a] Agui, Takeshi and Nagahashi, Hiroshi "A Description Method of Handprinted Chinese Characters", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-1 No 1, January 1979 Chinese handwriting recognition: features are concatenate, crossing, near, relative location among partial patterns. [Agui79b] Agui, Takeshi and Nagahashi, Hiroshi "A Coding Method of Chinese Characters", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-1 No 4, October, 1979 Chain codes, production rules, block codes, to encode Chinese characters as chain-code strings for comparison [AhoAV77a] Aho, A.V. and Ullman, J.D. "Principles of Compiler Design", Addison-Wesley, Reading, Massachusetts 1977 Compiler/programming language design: the "Dragon Book" [Aldefeld80] Aldefeld, B., Levinson, S.E., and Szymanski "A minimum-distance search technique and its application to automatic directory assistance", Bell Systems Technical Journal, Vol 59 No 8, pp , October 1980 Cited in Kahan87: : best method of organizing spelling dictionary for context in character recognition (actual paper uses spoken spelled speech recognition of letters) Context raises 80% on letters to 98.6% on names, 71% to 97.2% on name entries [AliF77a] Ali, F. and Pavlidis, T. "Syntactic recognition of handwritten numerals", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-7, 1977, pp [AndersonRH76] Anderson, Robert H. "Intelligent Terminals: A Potential Solution to Complexity in User-Network Interfaces", Proceedings of ASIS Annual Meeting, San Francisco, California, October 4-9, 1976 NTIS citation index [AndersonRH77a] Anderson, R. H. "Two-Dimensional Mathematical Notation", in "Applications of Syntactic Pattern Recognition", Springer Verlag, 1977, pp (partial copy) Top-down parser for 2D mathematical expressions handwritten on a tablet. Cites to Ledeen recognizer. [ApseyR78a] Apsey, R. "Human Factors of Constrained Hand-print for OCR", IEEE Trans. Systems, Man, and 8:26:16 PM]

50 ( ) History of Pen and Gesture Computing Cybernetics, Vol SMC-8 No 4, pp , April 1978 Constraints on handwriting using various "pre-printing" guides: dots, boxes, shapes, lines [ArakawaK78a] Arakawa, K., Odata, K., and Masuda, T. "On-line recognition of hand-written characters -- Alphanumeric, hiragana, katakana, kanji", Proc. 4th IJCPR, Kyoto, Japan, November 7-10, 1978, pp Published later as (Arakawa82) [AraziB78a] Arazi, B. "Handwriting Identification by Means of Run-Length Measurements", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-7 No 12, pp 878-ff, December 1977 off-line signature verification, study of vertical-line vs. horizontal-line run-length to do signature verification [ArmstrongWW79a] Armstrong, William W. and Gecsei, Jan "Adaptation Algorithms for Binary Tree Networks", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-9 No 5, May 1979, pp 276-ff adaptive tree networks, heuristic responsibility, with specialized sub-trees [Azure77] Azure, Leo L. "Electronic Notebook for Use In Data Gathering, Formatting, and Transmitting System", United States Patent 4,016,542, assigned to Azurdata, Inc., Richland, WA, April 5, 1977 Portable hand-held data-entry device, with radio link to a central computer. Uses a simple keyboard, does not refer to hand-written input [BabcockRT77] Babcock, R.T. "Simulation Method of Feature Selection for Unconstrained Handprinted Characters", M.S. Thesis, Department of Electrical Engineering and Computer Science, June 1977, Massachusetts Institute of Technology, Cambridge, Massachusetts 2-Z recognition: Blesser's and Shillman's group at Massachusetts Institute of Technology [Badler79] Badler, Norman I. and Smoliar, Stephen, W. "Digital Representations of Human Movement", Computing Surveys, Vol 11 No 1, March 1979 Translation of Labanotation dance notation to computer graphics of human movement: dance gestures [Baecker76] Baecker, Ronald "A conversational extensible system for the animation of shaded images", Proc. of 3rd Annual Conference on Computer Graphics and Interactive Techniques, Philadelphia, PA, pp Pen/lightpen system for interactive animation. Images, and their motion, sketched by user free-hand. Citation to Alan Kay presentation at 1972 ACM conference [Baecker79a] Baecker, Ronald, Buxton, William, and Reeves, William "Towards Facilitating Graphical Interaction: Some Examples from Computer-Aided Musical Composition", Proc. of 6th Man-computer Communications Conference, Ottawa CA, May 29-30, 1979 Musical notation recognition on digitizer [Baecker79b] Baecker, Ronald "Digital Video Display System and Dynamic Graphics", Computer Graphics, Vol 13 No 2, August 1979 Advantage of raster graphics terminal (e.g. video/tv) over random scan terminals (vector graphics): digital representation of image stored in computer, not drawn in storage scope on display. [Baecker80a] Baecker, R. "Towards an Effective Characterization of Graphical Interaction", in Methodology of Interaction, Guedj et al, editors, North-Holland Publishing Company, 1980, pp 127-ff User-interface: best user interface is device dependent, not device independent. Broad list of abstract qualities (type of feedback, etc.) of good user interfaces. Gesture-recognition for musical scores input. [Baecker80b] Baecker, Ronald "Human-computer interactive systems: A State-of-the-art review", in Processing of Visible Language 2, Kolers, Paul A., Wrolstad, Merald E., and Bouma, Herman, editors, Plenum Press, New York and London, 1980, pp User-interface review: trainable handwriting character recognizers: recognizer must be accurate and rapid Gesture-recognition for musical scores input [BakerJ80] Baker, Janet MacIver, Dialog Systems Inc. "Brief Status Summary for Automatic Speech Recognition at the start of the 80's", SAE Technical Paper Series , Soc. of Automotive Engineers, 400 Commonwealth Drive, Warrendale, Pennsylvania Report on speech recognition technologies, more than vendors and commercial systems, by founder of Dialog System (later Dragon) founder Janet Baker. Reports on *independent* testing of continuous speech / limited vocabulary system for digits and control words. [BarnardP76a] Barnard, P. and Wright, P. "The Effects of Spaced Character Formats on the Production and Legibility of Handwritten Names", Ergonomics, Vol 19 No 1, Jan 1976, pp (re box-less) wider character spacing makes stuff less readable Boxed input should match normal writing size (handwriting constraints) [BarrowHG77] Barrow, H.G., Tenenbaum, J.M., Bolles, R.C., and Wolf, H.C. "Parametric correspondence and chamfer matching: Two new techniques for image matching", Proc. 5th Int'l. Joint Conf. on Artificial Intelligence, August 1977, pp Also SRI International Technical Note 153 Cited in Burr81 Approximate matching/recognition for visual images of a landscape against a perspective/point-of-view generated image from landscape (topographic?) data, without requiring camera calibration in advance. Compare with structure-through-motion? 8:26:16 PM]

51 ( ) History of Pen and Gesture Computing [Bartram78] Bartram, D.J. "Post-iconic visual storage: Chunking in the reproduction of briefly displayed visual patterns", Cognitive Psychology, Vol 10, 1978, pp (abstract only) Cited in IchikawaS84 Subjects presented (flashed) sets of disks: remembered by groups/chunks. Is this where Buxton got his "chunking" ideas? [BeckerP77] Becker, Peter W. "Recognition of Patterns Using the Frequencies of Occurrence of Binary Words", Springer Verlag, Wien and New York, second revised edition first edition 1968, previously doctoral thesis: no physical copy, citation in IEEE. Use of spelling dictionary (with word frequency) for character/handwriting recognition. [BejczyAK79a] Bejczy, A.K. and Paine, G. "Transducer with a sense of touch", NASA Tech Briefs ISSN X, 4; 2; p 173, December 1, 1979 (abstract only) Matrix of pressure sensors for tactile (robotic) sensing: pressure map. Compare with multi-touch pressuresensing touchscreens. [BejczyAK80a] Bejczy, Antal K. "Sensors, Controls, and Man-Machine Interface for Advanced Teleoperation", Science, vol 208, Jun 20, 1980, pp Overview of electronic-skin touch sensors for teleoperation and robotics. Force-sensing (FSR?) "multipoint" (multi-touch) sensor grid/matrix, 2 mm resolution. Visual display of pressure map. Also describes slip (lateral force) sensors, using rolling ball mechanism. [BellDE76a] Bell, D.E. and La Padula, L.J. "Secure Computer System Unified Exposition and Multics Interpretation", Mitre Corporation, technical report ESD-TR / MTR-2997 Rev.1, March 1976 Bell-La Padula security model: based on security levels (MAC mandatory access controls) of Dept. of Defense. Part of Multic security analysis. read/write/execute/delete manners of use / access modes. [BernsteinMI76] Bernstein, M.I. "Interactive Systems Research: Interim Report to the Director, Advanced Research Projects Agency, for the Period 16 September 1975 to 15 March", 1976, System Development Corporation, Santa Monica, California, Report No SDC-TM-5243/005/00, 15 April 1976 NTIS citation index Speech understanding/recognition system [Berson77] Berson, Tom "Dynamic Handwriting Recognition by Computer", Ph.D. Thesis, University of London, 1977 [Berthod79] Berthod, M. and Maroy, J.P. "Learning in Syntactic Recognition of Symbols Drawn on a Graphic Tablet", Computer Graphics and Image Processing, Vol 9, 1979, pp The main problem in handwriting recognition is wide range of variability Big practical problem of on-line character recognition (vs OCR) is large deformations in handwriting Statistical approaches: complex classifying scheme and meaningless features Problem of statistical recognition is features do not correspond to any visual entity Says on-line recognition should allow added new symbols (writable icons for commands) (?) Character recognition needs techniques adapted to characters, not perception of drawings in general Character recognition should use human-meaningful concepts "chain codes" of straight line, plus/minus curve, pen-lift, cusp Two-level chain code, decision tree like our BLRTs, nodes (but adaptive?) Gives a grammar of permissible sequences for a sort of chain code [Berthod80] Berthod, M. and Ahyan, S. "On line cursive script recognition: A structural approach with learning", Proceedings of the 5th International Conference on Pattern Recognition, Miami Beach, Florida, December, 1980, pp Cited in Bozonivic89 Cited in Bozinovic89 as doing adaptive/training handwriting recognition, but with syntax of letter formation and writing dynamics for strokes [BestRM80a] Best, Robert M. "Preventing software piracy with crypto-microprocessors", Proc. IEEE Spring COMPCON '80, San Francisco Feb , 1980 pp Early paper on blocking software piracy by encrypting software as distributed, or other media on videodisks, decrypted by secure system (trusted system) processor. Decoding is one instruction at a time (which makes decoding easier to hack), secure hardware. Cited in Tyger/Dyad for secure co-processors. [Bezdek77] Bezdek, J.C. and Castelaz, P. "Prototype classification and feature selection with fuzzy sets", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-7 No 2, 1977, pp Cited in Biswas81 [Bjorklund77] Bjorklund, C.M. "Syntactic analysis and description of stroke-based shapes", Proceedings of IEEE Pattern Recognition and Image Processing Conference, Troy, New York, June 6-8, 1977, pp [BlesserB76a] Blesser, B., Kuklinski, T.T., and Shillman, R.J. "Empirical Tests for Feature Selection Based on a Psychological Theory of Character Recognition", Pattern Recognition, Vol 8, pp 77-85, 1976 How we pick functional attributes: compare with other phenomenological articles, confusion matrix. [BobickTW78a] Bobick, Thomas W. "Capacitive touch-activated transducer system including a plurality of oscillators", United States Patent 4,103,252, July 25, 1978 Capacitive touchpad/arrow keys using four electrodes, finger proximity changes timing of RC oscillator circuits in each electrode. [Boldridge77] Boldridge, A.G. and Freund, R.W. "Personal Identification Apparatus", United States Patent 8:26:16 PM]

52 ( ) History of Pen and Gesture Computing 4,035,768, assigned to Veripen Incorporated, New York, New York, July 12, 1977 Signature verification using pressure (actually, tip force) [Bolt80] Bolt, R.A. "'Put-That-There': Voice and Gesture at the Graphics Interface", SIGGRAPH '80 Proceedings, published as Computer Graphics, Vol 14 No 3, July 1980, pp User interface combining pointing, hand gestures, and voice recognition. transparent touch-sensitive screen overlays (touchscreen); touch-sensitive keymap (virtual keyboard). See also Schmandt 1980 thesis. [BorningA79] Borning, Alan "ThingLab -- A Constraint-Oriented Simulation Laboratory", Ph.D. thesis, CS Dept. Stanford University, published as SSL-79-3 July 1979 editing of graphical drawings/electronic ink, no gesture recognition [Bouchard80] Bouchard, D.C. and Toussaint, G.T. "Heuristic search methods for efficient use of dictionary information in text recognition", School of Computer Science, McGill University, Technical Report SOCS 80.5, May, 1980 Cited in HullJJ83 [Brayton79] Brayton, Robert K. "Character Recognition System and Method Multi-Bit Curve Vector Processing", United States Patent 4,177,448, assigned to IBM Corp, Armonk, New York, December 4, 1979 Setting up character recognition based on large character training set Shapes are canonicalized to standardized/normalized pieces of curves/segments [Bresenham77] Bresenham, J.E. "A Linear Algorithm for Incremental Digital Display of Circular Arcs", CACM, Vol 20 No 2, February 1977, pp Circle-drawing algorithm on bit-mapped displays: cites algorithms for other curves [Bridle79] Bridle, J.S. and Brown, M.D. "Connected word recognition using whole word templates", Proceedings of the Institute for Acoustics, 1979, pp Cited in Kruskal83 [Briem79] Briem, G.S.E. "Wanted: Handwriting That Fits Modern Pens", Visible Language, Vol XIII No 1, 1979, pp Background on handwriting variability effects from stylus design, for physical writing pen instrument, e.g. nib pen versus ballpoint, etc. [BrownMD76a] Brown, Maxine D. and Smoliar, Stephen W. "A graphics editor for labanotation", ACM SIGGRAPH Computer Graphics Newsletter, Vol 10 Issue 2, Summer 1976, pp Graphical editor user interface for labanotation -- written notation for dance movements. Prototype used keyboard and graphical terminal only, plans to add tablet input (with gestures?) [BrownMK80a] Brown, M.K and Rabiner, Lawrence R. "An Adaptive, ordered, graph search technique for dynamic time warping for isolated word recognition", IEEE Trans. Acoustics, Speech and Signal Processing, Vol 30, 1982, pp Cited in Schomaker86 [BrownMK80b] Brown, M.K. and Ganapathy, S. "Cursive Script Recognition", 5th International Conference on Character Recognition, 1980, pp Cursive script recognition: uses training data-base intentionally with lots of variability [BurkeB76] Burke, Barbara "Reading Writing", M.I.T. Reports on Research, Vol 4 No 3, November 1976 Report on work of Shillman, Blesser, and Kuklinski [Burr79] Burr, D.J. "A technique for comparing curves", Proceedings of the IEEE Conference on Pattern Recognition and Image Processing, Chicago, 1979, IEEE New York, pp Cited in Kruskal83 [Buxton79] Buxton, William; Sniderman, Richard; Reeves, William; Patel, S.; and Baecker, Ronald "The evolution of the SSP score editing tools", Computer Music Journal, Vol 3, 1979, pp Reprinted in "Foundations of Computer Music", MIT Press, Cambridge MA 1985, pp music-editing notation, single/uni-stroke handwriting character recognition Shows other GUIs for music input and editing other than recognition Cited in Wolf87a [Calvert80] Calvert, T.W., Chapman, J., and Patla, A "The Integration of Subjective and Objective Data in the Animation of Human Movement", Conference Proceedings: ACM /80/ Labanotation: dance gestures [Canada78] CSA "Numeric Character Set for Hand-printing", Canadian Standards Association, Standard Z M1978, February 1978 Canadian equivalent of ANSI74 hand-print standard to constrain handwriting for OCR [Carau78] Carau, F.P. "Easy-to-use, High-Resolution Digitizer Increases Operator Efficiency", Hewlett-Packard Journal, December 1978, pp 2-13 Electrostatic Digitizer measuring X and Y sequentially in time causes line to bow due to velocity-related errors. Gives first-order correction. Numerous descriptions of other digitizer position errors, such as nonorthogonal axes, and how to correct for them. [CardSK78a] Card, Stuard K.; English, William K.; and Burr, Betty J. "Evaluation of Mouse, Rate-Controlled 8:26:16 PM]

53 ( ) History of Pen and Gesture Computing Isometric Joystick, Step Keys, and Text Keys for Text Selection on a CRT", Ergonomics, Vol 21 No 8, 1978 pp Mouse faster for text pointing than joystick, arrow keys, or keys. Cites other work on Grafacon tablet, not included in this study. [Carvey76] Carvey, P.C. "Electrographic System", United States Patent 3,975,592, assigned to Applicon, Incorporated, Burlington, Massachusetts, August 17, 1976 Digitizer using coarse, then fine scanning [CatmullE78a] Catmull, Edwin "A Hidden-Surface Algorithm with Anti-Aliasing", Proc. SIGGRAPH '78 5th Annual Conf. on Computer graphics and interactive techniques, pp Early paper related to alpha blending, applied to anti-aliasing in computer graphics. [Cederberg78] Cederberg, R.L.T. "An iterative algorithm for angle detection on digital curves", Proceedings of 4th International Joint Conference on Pattern Recognition, Kyoto, Japan, November 7-10, 1978, pp [ChangP76] Chang, P. "Efficient Keyboard Layouts for Inputting Chinese Characters", B.S. Thesis, Department of Electrical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts May confusions/ambiguous character in defined Chinese/Kanji even if every basic stroke has a unique keyboard stroke (due to positions of strokes?) [ChangS77] Chang, S. and Nagy, G. "Deposit-Slip-First Check Reading", IEEE Trans. Systems, Man, and Cybernetics, January 1977, pp Operations in OCR: bank can control deposit slip format, but not checks [ChaoYM77a] Chao, Yao-Ming "Recognition of handwriting Chinese Characters", Student Paper, M.I.T., Fall 1977 direction codes / chain-codes, Chinese characters recognition [ChenCH77a] Chen, C.H. "Statistical Pattern Recognition - Review and Outlook", Systems, Man, and Cybernetics Review, Vol 6 No 4, August, 1977, pp 7-8 Feature extraction remains key problem in pattern recognition: features must be based on problem at hand. Context aids statistical pattern recognition. [ChiangA79a] Chiang, Anne; Curry, Don; and Zarzycki, Mark "A Stylus Writable Electrophoretic Display Device", SID Symposium Digest, 1979, pp Flexible tablet/touchscreen with ITO or SnO2 on transparent plastic. Electrostatic stylus leaves ink directly on (character) display, can be erased by reversing voltage in the stylus (or eraser). Display only, no position sensing(?). [Chinnuswamy80] Chinnuswamy, P. "Recognition of handprinted Tamil characters", Pattern Recognition, Vol 12, 1980, pp Cited in Loy82 on Indian handwriting script (Tamil) [ClarkJH76a] Clark, James H. "Designing Surfaces in 3-D", CACM,Vaol 19 No 8, August, 1976, pp Input of 3-D surfaces using control points and spline fitting for surfaces. Three-dimensional digitizer is electromechanical: wand with three filament lines attached to end, sensors read how much line is paid out in for all lines and computes X/Y/Z position. Head-mounted stereoscopic display. Cites to Lincoln Wand. [CoffinS78a] Coffin, Stephen "Spatial frequency analysis of block letters does not predict experimental confusions", Perception and Psychophysics, Vol 23 No 1, 1978, pp Human recognition of letters uses features, not mathematics: Fourier spatial frequency Refers to edge and line detection/recognition in human vision, then Position as something different from low-level features Criticizes studies of human recognition that had visual distraction [Computerwoche77] Computerwoche "Online-DE mit dem Bleistift: Datenerfassung als Nebensache", Computerwoche 21/1977, May 20, 1977 Quest Automation Datapad "Datapad puts your pencil Online to a computer". Dorset U.K. Handwriting recognition input on a paper form, 32-character LED display, uses Nova 16-bit minicomputer as controller. Statement is that data entry is done just on the side automatically while person is filling out regular paper form. [Computerwoche80a] Computerwoche "Quest Automation Ltd.: Handschriftleser für Direkterfassung", Computerwoche 16/1980, April 18, 1980 Quest Automation Datapad/Micropad, newer version of Datapad: 40-character display. Says less than one hour training required. Reference to booth at CeBIT exposition. [Computerwoche80b] Computerwoche "Statt über Keyboard mit der Hand: Micropad gibt Sauklaue ein", Computerwoche 10/1980, March 7, 1980 Quest Automation Datapad/Micropad handwriting tablet / keyboard replacement. [CooperLN76] Cooper, Leon N., and Elbaum, Charles "Information Processing System", United States Patent 3,950,733, assigned to Nestor Associates, New York, New York, April 13, 1976 Nestor patent: neural net, adaptive memory [CooperLN77] Cooper, Leon N., and Elbaum, Charles "Information Processing System", United States Patent 8:26:16 PM]

54 ( ) History of Pen and Gesture Computing 4,044,243, August 23, 1977, assigned to Nestor Associates, Stonington, Connecticut Nestor Graphics patent [CoxCH76] Cox, C.H. III and Coueignoux, P. "Concise Letter/Type Font Description: Theory and Computer Implementation", Internal report, Massachusetts Institute of Technology Industrial Liaison Program, Department of Electrical Engineering and Computer Science, November 20, 1976 OCR variability based on thinned "skeletons", generatively similar to chain-codes. Rules for line thinning, seraphs for typed characters. [CoxCH78] Cox, C., Blesser, B. and Eden, M. "Graphical Context of Printed Characters", Visible Language, Vol XII No 4, Autumn 1978, pp Generative variability: points out existence of "rules" for how to write, even if not conscious [Crane77a] Crane, H.D., and Savoie, R.E. "An On-Line Data Entry System for Hand-Printed Characters", IEEE Computer, pp 43-50, March 1977 On-line handwriting/shape recognition: Uses ULDR chain codes, similar to BLRT chain codes: accelerometer pen? [Crane77b] Crane, H.D., Ostrem, J.S., Wolf, D.E., and Wang, T.N.C. "A Technique for the Input of Handprinted Chinese Characters Based on Sequential Stroke Recognition", Proceedings of International Computer Symposium 1977, Taipei, Republic of China, Vol 1, pp Describe drag force, not acceleration, of SRI pen Kanji Chinese has Gaussian distribution of number of strokes, with 11 average 881 characters in basic Kanji defined by Japanese ministry of education Problem focuses on confusions, since did not have a large enough data base to get valid statistical results on recognition accuracy [Crane77c] Crane, H.D, Wolf, D.E., and Ostrem, J.S. "The SRI pen system for automatic signature verification", Symposium Proceedings NBS Trends and Applications 1977, Gaithersburg, May 1977, pp Liu,CN79 Obscured tip on SRI pen. SRI dynamic pen design: see Crane75. [CraneHD79a] Crane, H.D. and Wolf, D.E. "Dynamic Re-creation of Signatures", United States Patent 4,156,911, assigned to Stanford Research Institute, Menlo Park, California, May 29, 1979 Signature verification by human eyeball: patent on compressing signature to minimum data without losing detail Use of local extrema/chain codes to store written signatures with minimum memory [CraneHD79b] Crane, H.D. and Wolf, D.E. "Handwritten signature verification system", United States Patent 4,086,567, April 25, 1978, assigned to Stanford Research Institute, Menlo Park, California Signature verification by looking for landmarks (pen lifts) Hew Crane: compressed-data method of storing signatures for human reading [CraneHD80a] Crane, H.D. and Wolf, D.E. "Signal Train Verification System Using Landmarks", United States Patent 4,190,820, February 26, 1980, assigned to Stanford Research Institute, Menlo Park, California Elastic/rubbery matching of signature to segment into sections to be stretched or contracted for template matching Landmarks in signatures which are invariant features [CranmerTV77a] Cranmer, Terrence, V., Mary "Braille display switching matrix and circuit utilizing the same", United State Patent 4,037,200, July 19, 1977 Braille clock display with (fixed) braille dots on insulated conductive pins. Braille characters in vertical columns. Each pin is a touch-controllable non-mechanical switch (by short circuit to surrounding touch substrate: compare with resistive touchscreen) for a set of the terminals single braille character). multitouch? [Crawshaw77] Crawshaw, Martin, and Ottaway, Mary "A Contact-Pencil for Research on Writing", Quarterly Journal of Experimental Psychology, Vol 29, 1977 pp Stylus using graphite/pencil lead to sense pen down/up. [CrowFC77a] Crow, Franklin C. "Shadow Algorithsm for Computer Graphics", Proc. SIGGRAPH '77, July , San Jose, pp Computational techniques for simulated shadows in virtual images: silhouette edges of polygonization. Does not mention ray-tracing. [CrowleyMillingM77a] Crowley-Milling, Michael "How CERN broke the software barrier", New Scientist, 29 September 1977, pp CERN computer science innovations born of necessity: interpretive language with interactive debugging with real-time control, virtual machines matched well to multiple and distributed computers, invention of touch screen with structured menus -- now available commercially -- and "rolling ball" (trackball). [DIN77] Deutsches Institut für Normung e.v. "Schrift B für die maschinelle optische Zeichenerkennung", DIN , September, 1977 German standard for OCR readable handwritten characters, similar to ANSI standard, but lists several as "not intended for OCR" (too hard to recognize?) [Dasarathy78] Dasarathy, B.V. and Kumar, K.P.B. "Chitra: cognitive handprinted input-trained recursively analyzing system for recognition of alphanumeric characters", International Journal of Computing and Information 8:26:16 PM]

55 ( ) History of Pen and Gesture Computing Science, Vol 7, 1978, pp Cited in Li89 [DavisLS76] Davis, Larry S. and Rosenfeld, A. "Applications of relaxation labeling: Spring-loaded template matching", Proceedings of 3rd International Joint Conference on Pattern Recognition, 1976, pp Davis,LS77 [DavisLS77] Davis, Larry S. "Understanding shape: Angles and sizes", IEEE Trans. Computers, Vol C-26, March 1977, pp Cited in Teh89 [DavisLS79] Davis, Larry S. "Shape Matching using Relaxation Techniques", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-1 No 1, January 1979, pp Relaxation process for matching shapes (coastlines) by figures of merits on pairs of angles (segment matching) Cited in Baird84: on template matching in feature space using minimum distance [DavisRL77a] Davis, Robert L and Howells, Joseph A. "Position Determining Apparatus and Transducer Therefore", United States Patent 4,012,588, March 15, 1977 Acoustic digitizer using multiple pick-up "sonar" microphones from sound transducer on stylus: SAC Science Accessories Corporation [DiffieW76a] Diffie, Whitfield and Hellman, Martin E. "New Directions in Cryptography", IEEE Trans. on Information Theory, Vol IT-22, No 6, November 1976, pp General description of public key cryptography, cryptographic (public key) signatures. Authentication by centrally-stored Public File, cryptographically signed. [DiffieW76b] Diffie, Whitfield and Hellman, Martin E. "Multiuser cryptographic techniques", Proc. NCC, 1976, pp General description of public key cryptography/authentication. Private conversations between any two individuals without central key management. Digital signature has non-repudiation (by signer), and unforgeable (by recipient). [Donahey76] Donahey, Alvin V. "Character Recognition System and Method", United States Patent 3,996,557, December 7, 1976, assigned to MI-squared Corporation, Columbus, Ohio On-line character recognition using vertical conductive bar zones/region Says does script, but it is not connected writing script, only for separate/discrete characters. [DonelsonWC77a] Donelson, William C. "Spatial Management of Data", M.S. Thesis, Massachusetts Institute of Technology, September 1977, Department of Architecture See Donelson78 [DonelsonWC78a] Donelson, William C. "Spatial Management of Information", ACM SigGraph Conference, 1978, pp Gesture (hand-gesture?) user interface for viewing data using large-screen display. Summagraphics magnetostrictive digitizer tablet with stylus for writing annotation (electronic ink?), Joypad touchscreen (resistive tablet) for user input? Both verbal and electronic ink annotations: how edit? See also Donelson Master's thesis. [DosterW77] Doster, W. "Contextual postprocessing system for cooperation with a multiple-choice character recognition system", IEEE Trans. Computers, Vol C-26, 1977, pp Spelling dictionary look-up for context correction: in separate character OCR (handwriting) Multiple-Choice Single Character Recognition System and Contextual Postprocessing System [DosterW80] Doster, W. and Schuermann, J. "An application of the modified Viterbi-algorithm used in text recognition", Proceedings of 5th International Conference on Pattern Recognition, 1980, pp In Tappert's bibliography [Driscoll77] Driscoll, R.J. et al "Real-Time Reading of Handwritten Symbols and Applications", Proceedings of the Conference eon Digital Processing of Signals in Communications, September 9, 1977, Institute of Electronic and Radio Engineers, pp Cited in Sklarew patents [Druse78] Druse, B. and Rao, C.V.K. "A matched filtering technique for corner detection", Proceedings of the 4th International Joint Conference on Pattern Recognition, Kyoto, Japan, November 7-10, 1978, pp [Du80] Du, H.C. and Lee, R.C.T. "Symbolic Gray Code as a Multikey Hashing Function", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-2 No 1, pp 83-90, January 1980 Very fast searching using keys (brute-force engineering) [Duerr80] Duerr, B., Haettich, W., Troph, H., and Winkler, G. "A combination of statistical and syntactical pattern recognition applied to classification of unconstrained handwritten numerals", Pattern Recognition, Vol 12, 1980, pp Cited in Kerrick88 [Dumas77] Dumas, G. "A Dual Microprocessor Applications: The Alphabec-75 Data Capture System", in "Microcomputer Design and Applications", Academic Press, New York/San Francisco/London, 1977, pp :26:16 PM]

56 ( ) History of Pen and Gesture Computing Mentions user learning/adaptation/training times and success rate: some learned fast, some never did (!) Xebec/SRI/Crane original character recognition pen product using chain codes and an accelerometer in the writing pen [DunnRankin78] Dunn-Rankin, Peter "The Visual Characteristics of Words", Scientific American, January 1978, pp Word recognition in humans (graphical word context). Other Dunn-Rankin papers describe inter-letter confusion in human perception of English alphabet: compare with Functional Attributes of Shillman [Duvernoy79] Duvernoy, J. and Charraut, D. "Stability and stationarity of cursive handwriting", Pattern Recognition, Vol 11, 1979, pp In Tappert's bibliography Handwriting variability in cursive script [DymH78a] Dym, Herbert and Krakinowski, Morris "Capacitive two dimensional tablet with single conductive layer", United States Patent 4,087,625, May 2, 1978 Charge-ratio capacitive/electrostatic tablet, triangular electrodes coming in from two sides. Measures capacitive coupling with AC signal. [EachusJJ77a] Eachus, Joseph J. and Graff, Theodore S. "Touch Sensitive Device", United States Patent 4,055,735, October 25, 1977 Touch-button (?) panel/tablet with grid of pressure sensitive material on two sides of a printed circuit board substrate. Cross-wire construction: compare with Kaplow? [EerNisseEP77a] EerNisse, E.P., et al "Piezoelectric Sensor Pen for Dynamic Signature Verification", Conference 1988 Int. Electron Devices Meeting, Washington, D.C., December 1977, pp Cited in Chainer85a Signature verification using pen/stylus with piezoelectric pressure/force sensors for writing pressure, also lateral shear force [EerNisseEP78a] EerNisse, E.P., Land, C.E., and Snelling, J.B. "Input Apparatus for Dynamic Signature Verification Systems", United States Patent 4,078,226, assigned to the United States of America, March 7, 1978 Signature verification stylus pen using piezoelectric force/pressure sensors to sense X and Y forces/acceleration [EhrichRW76] Ehrich, Roger W. "A Readily Computable Decision Rule with Variable Dimensionality", IEEE Trans. Computers, May, 1976, pp Bayes and Meyman-Pearson statistical classifiers to select likelihood ratios Features statistically best are generally "specialists" in distinguishing that class [EhrichRW78] Ehrich, Roger W. "Handwriting Recognition", Encyclopedia of Computer Science, Vol 9, 1978, Belzer, Holzman and Kent, editors, Dekker, 1978, pp [ElectronicsAndPower79a] Electronics and Power "A British world first in data entry - Micropad Ltd., Quest Automation Lt.", Electronics and Power (UK), October 1979, p. 685 Quest Micropad / Datapad handwriting terminal announcements -- see Micropad files. [Elographics80] Gibson, W., Talmage, J. "Nonplanar Transparent Electrographic Sensor", United States Patent 4,220,815, September 2, 1980, assigned to Elographics, Incorporated, Oak Ridge, Tennessee Resistive-sheet/film digitizer using small spacing dots between a conductive and a resistive layer: Elographics [Engdahl77] Engdahl, Jean "Data Entry and Decoding System for Scripted Data", United States Patent 4,005,400, January 25, 1977, assigned to Societe Suisse pour l'industrie Horologere Management Services S.A., Bienne, Switzerland Four-segment panel for character recognition, zone-based recognition for handwriting, using a stylus [EngelbrechtRS76a] Engelbrecht, R.S. "Handwriting Identification Technique", United States Patent 3,962,679, June 8, 1976, assigned to RCA Corporation, New York, New York Signature verification using writing speed/velocity [ErmanLD80a] Erman, Lee D.; Hayes-Roth, Frederick; Lesser, Victor R.; and Reddy, D. Raj "The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty", ACM Computing Surveys, Vol 12 No2, June 1980 pp speech-recognition/speech-understanding system, review of similar systems. Solution crosses multiple problem domains: phoneme detection, word segmentation, vocabulary, grammar, problemdomain/discourse semantics. Describes technique as "blackboard" system. [Farag79] Farag, R.F.H. "Word-Level Recognition of Cursive Script", IEEE Trans. Computers, Vol C-28 No 2, February 1979, pp Sketch recognition of whole script words vs a table/word dictionary for context Testing involved only a very small sample (ten words) [FeldmanSI79] Feldman, S.I. "Make -- a Program for Maintaining Computer Programs", Software Practice and Experience, Vol 9 No 4, pp , 1979 Overview of Unix MAKE build tool with dependencies [FenclGA80a] Fencl, George A. "Low Noise System and Method for Sequentially Sensing Induced Signals in 8:26:16 PM]

57 ( ) History of Pen and Gesture Computing digitizer Grid Conductors", United States Patent 4,185,165, January 22, 1980 Electromagnetic tablet digitizer: states that conductive marks (?) on form can affect performance [Feucht77] Feucht, Dennis "Pattern Recognition: Basic Concepts and Implementations", Computer Design, December 1977, pp Overview of pattern recognition: decision functions, multicategory classifiers (type I: simple, type II: pairwise, type III: extended to all classes), hyperplanes and weight vectors in feature space, minimumdistance classifiers Clustering techniques: maximin-distance, K-Means algorithms, general cluster-seeking algorithm. Learning-adaptive recognition: perceptron, absolute increment, multicategory perceptron. [FisherE76] Fisher, E. "The use of context in character recognition", COINS Technical Report (Ph.D. thesis, Computer and Information Sciences Department, University of Massachusetts, Amherst), 1976 Cited in DosterW77 on spelling correction dictionary? Morse code recognition [Fitzgerald77] Fitzgerald, William Joseph, Shelton Jr., Glenmore Lorraine, and Wolfe, Robert Nolan "System for converting a rough sketch to a finished drawing", United States Patent 4,058,849, assigned to International Business Machines Corp, New York, New York, November 15, 1977 Editing of handwritten sketches by making the lines more nearly vertical and horizontal, and moving the endpoints of the lines: preserves linkages among strokes Also shows use of an overlay keyboard, presenting a virtual keyboard on the tablet surface Cites Eichen73 patent, and "Instant Draftsman" article from 1972 [FoleyJD80a] Foley, James D; Chan, Peggy; and Wallace, Victor L. "The Human Factors of Graphic Iteration: Tasks and Techniques", Tech. Report 508, US Army Research Institute for the Behavioral and Social Sciences, December 1980 Study of user interface input devices for graphical applications: light pen, buttons/keyboard, chord keyboard, speech recognizer, tablet, mouse, joystick. Tablet most frequently used device for pointing input (1980). [Foster80] Foster, D.H. and Mason, R.J. "Irrelevance of local position information in visual adaptation to random arrays of small geometric elements", Perception, Vol 9, 1980, pp (BLRT chain codes) relative position not as significant as local features in human recognition [FranklinRC78a] Franklin, Robert C. and Fuller, Rank I. "Electronic Wall Stud Sensor", United States Patent 4,099,118, July 4, 1978 Interesting example of electrostatic/capacitive sensing: compare with electrostatic touchscreens and digitizers. Detects change in dielectric constant in proximity, rather than higher or lower dielectric constant [Fraser80] Fraser, A. "Comments on 'A Study of Man-Machine Interaction Problems in Character Recognition'", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-10 No 9, September 1980, page 589 Says Suen is full of it Says variability in writing styles is caused by deep, fundamental human individuality Points out "obvious" testing bias in Suen's paper A graphologist speaks out on graphonomics handwriting research Significant comment on dynamic character recognition by a graphologist "neat", constrained handwriting is not realistic [Freeman77] Freeman, H., and Davis, L.S. "A Corner-Finding Algorithm for Chain-Coded Curves", IEEE Trans. Computers, Vol C-26, pp , March 1977 Adds "corner" detection to chain codes Source of Barry's original "hi-curve" algorithm for corners? [Freeman78a] Freeman, H. "Application of the generalized coding scheme to map data processing", Proceedings of IEEE Pattern Recognition and Image Processing Conference, Chicago, Illinois, May 31-June 2, 1978, pp [Freeman78b] Freeman, H. and Saghri, A. "Generalized chain codes for planar curves", Proceedings of the 4th International Joint Conference on Pattern Recognition, Kyoto, Japan, November 7-10, 1978, pp [Frey77] Frey, P.W. "Chess Skill in Man and Machine", Springer Verlag, New York, 1977 Survey of artificial intelligence ideas in computer chess [Fu77] Fu, K.S. "A Brief Review of Pattern Recognition", Systems, Man, and Cybernetics Review, Vol 6 No 4, August, 1977, pp 3-5 Equivalence of decision-theoretic/statistical and syntactic pattern recognition [FuKS80a] Fu, K.S. "Recent development in pattern recognition", IEEE Trans. Computers, Vol C-29, October, 1980 Cited in Fu86/Fu83 [FujimotoY76a] Fujimoto, Y., Kadota, S., Hayashi, S., Yamamoto, M., Yajima, S. and Yasuda, M. "Recognition of Hand-printed Characters by Nonlinear elastic Matching", Proceedings of the 3rd International Joint Conference on Pattern Recognition, November 1976, pp Quotes Blesser, but says they never built an actual system Uses eight-direction vector/length chain codes for elastic matching Compare to Tappert's paper on elastic matching Goshtasby88 cites that this system used continuity (continuation marks and hooks between characters) to get 99.7% recognition accuracy [FujitaT76a] Fujita, T., Nakanishi, M., and Miyata, K. "The recognition of Chinese characters (kanji) using time variation of peripheral belt patterns", Proceedings of 3rd International Joint Conference on Pattern Recognition, 8:26:16 PM]

58 ( ) History of Pen and Gesture Computing Coronado, California, November 1976, pp [Gaertner80] Gaertner, K.P. and Holzhausen, K.P. "Controlling air traffic with a touch sensitive screen", Applied Ergonomics, Vol 11 No 1, 1980, p Cited in Beringer89: user-interface on electronic ink, integrated tablet/display. A clamp-on mechanical digitizer [GainesRS80a] Gaines, R. Stockton; Lisowski, William; Press, S. James; and Shapiro, Norman "Authentication by Keystroke Timing: Some Preliminary Results", Rand Corp. NSF Report R-2526-NSF, May 1980 User authentication using timing digraphs from user's keyboard input (biometrics, similar to signature authentication/verification). Digraphs found effective, simple measurement. In particular frequent digraphs such as in, io, no on, ul. Only tested with experienced typists. See also Saltzer [Genesereth79] Genesereth, M.R. "The Use of Semantics in a Tablet-Based Program for Selecting Parts of Mathematical Expressions", MACSYMA User's Conference, V.E. Lewis, Ed. Citation by Soiffer Kajler on citeseer.nj.nec.com/context/906109/0 mentions use of GUI for circling parts of mathematical expressions, using semantic information [Geographics80] GeoGraphics "GeoGraphics Drafting Board Digitizer", GEOGRAPHICS Incorporated, 1318 Alms Drive, Champaign Ill , 1986 Drafting board digitizer: usable for whiteboard? [GibsonWA80a] Gibson, William A. and Talmage, John E. jjr. "Nonplanar transparent electrographic sensor", United States Patent 4,220,815, September 2, 1980 Resistive sheet transparent touchscreen digitizer, spacer dots keep the layers apart until pressed (Elographics) [GilmoreGC79a] Gilmore, G.C., Hersh, H., Caramazza, A. and Griffin, J. "Multidimensional letter similarity derived from recognition errors", Perception and Psychophysics, Vol 25, 1979, pp Experimental study of confusion matrix with human subjects of pairs of characters, and which features appear to be most relevant to confusion. Compare with Shillman phenomenological attributes? Only suggest results as a guide to picking similar characters, not for determining distinguishing features. Cited in IchikawaS84 [Gould78] Gould, J.D. "How experts dictate", Journal of Experimental Psychology: Human Perception and Performance, Vol 4, 1978, pp Cited in Martin89 [Greenaway78] Greenaway, David Leslie "Apparatus and Method for Producing an Electrical Signal Responsive to Handwriting Characteristics", United States Patent 4,122,433, assigned to LGZ Landis and GYR Zug Ag, Zug, Switzerland, October 24, 1978 Pen stylus moves across a ridged/quadrangled surface while writing: resulting vibrations give and indication of pressure force and writing velocity [Greenberg77] Greenberg, M.E. "An investigation of handwritten numeral character recognition", M.S. Thesis, Department of Electrical Engineering and Computer Science, August 1977, Massachusetts Institute of Technology, Cambridge, Massachusetts Throws out data from IEEE handwriting variability Database as "poorly written" [GroenFCA78a] Groen, F.C.A. and Verbeek, P.W. "Freeman-code probabilities of object boundary quantized contours", Computer Vision, Graphics, and Image Processing, Vol 7, 1978, pp Representation of contours (of shapes) using eight-direction Freeman chain codes. Cited in Teh89 [Grossman78] Grossman, Robert "Hybrid pressure/force transducers give robots a sense of touch", EDN, August 5, 1978, pp [GroverDJ79] Grover, D.J. "Graphics Tablets - A Review", Displays, July 1979, pp 83-93, IPC Business Press, Great Britain Attempt at complete classification of methods of graphical/tablet interaction. Broad overview of digitizer tablet technologies. [GrummerWL80a] Grummer, William L. and Hendriks, Hendrik B. "Capacitive Touch Switch Array", United States Patent 4,233,522, November 11, 1980 Capacitive touchscreen using grid/array of capacitive sensors. [Guberman76a] Guberman, S. and Litvin, Y. "Implementation of an Algorithm for Recognition of Handwritten Letters", Institute of Applied Mathematics, Academy of Sciences, USSR, TR-22, Moscow USSR (in Russian), 1976 Cited in Goodale83 [Guberman76b] Guberman, S. and Rozentsveig, V. "An Algorithm for Reading Handwritten Letters", (Automation and Remote Control), Vol 37, pp 751 ff Cited in Goodale83 [Haberman76] Haberman, W. and Fejfar, A. "Automatic Identification of Personnel through Speaker and Signature Verification -- System Description and Testing", Proceedings of Carnahan Conference on Crime Countermeasures, May 1976, pp :26:16 PM]

59 ( ) History of Pen and Gesture Computing [HallPAV80a] Hall, P.A.V. and Dowling, G.R. "Approximate string matching", Computing Surveys, Vol 12 no 4, December 1980, pp Cited in Srihari83: on context/spelling correction [Hanaki76] Hanaki, Shin-Ichi, Temma, Tsutomu and Yoshida, Hiroshi "An On-line Character Recognition Aimed at a Substitution for a Billing Machine Keyboard", Pattern Recognition, Vol 8, pp 63-71, January 1976 User-interface: early (1976) Nippon Electric prototype Katakana/Hiragana/Romanji product for data-entry: features chosen by "human intuition" Describes what appears to be oversampling, resampling, noise filter on digitizing tablet (Tel product) Resampling of handwritten tablet points to be minimum number of feature points for reconstruction/recognition: also used for data reduction/compression of line drawings Character segmentation by stroke time-out, writing in boxes, recognition result Shows shading pattern for boxes to write in 98% recognition rate, using decision tree node Refers to handwriting character segmentation based on recognition result/plausibility [HansonAR76a] Hanson, A.R., Riseman, E.M., and Fisher, E. "Context in Word Recognition", Pattern Recognition, Vol 8, 1976, pp (abstract only) Electronic copy is just abstract: see paper file. Low character error rates can give high word error rates. Use context on words to correct errors on single characters. "many theoretical systems are impractical in application". Cites use of "clear text" in context: but many applications are not (data entry) Tri-gram statistical Markovian context probabilities, not linguistic grammar context Cites problems of small training sets (98 characters per class) High (context) correction rates introduce many substitution errors The number of substitution errors is about the same as reject recognition Context after the recognition is done, not before [HardenJC77a] Harden, John Charles and Mastrangelo, Sebastian V.R. "Pressure-sensitive elastic resistor compositions", United States Patent 4,028,276, June 7, 1977 Pressure-sensitive elastomeric conductive rubber, one application is keyboard switches. Metallic-conductive particles in elastomer. [Hattich78] Hattich, W., Tropf, H., and Winkler, G. "Combination of statistical and syntactical pattern recognition - - Applied to classification of unconstrained handwritten numerals", Proceedings of Fourth International Conference on Pattern Recognition, Kyoto, Japan, November 7-10, 1978, pp Cited in Kahan87 [HayesRothF77a] Hayes-Roth, Frederick and Lesser, Victor R. "Focus of Attention in the HEARSAY-II Speech Understanding System", Proc. Fifth Int'l. Joint Conf. on Artificial Intelligence, Volume 1, IJCAI, pp speech recognition / speech understanding system, multiple knowledge sources (phonetic, segmentation, syntax, grammar, problem-domain-specific semantics) included in blackboard of current hypothesized results and alternatives. [HerbstNM76a] Herbst, Noel M., and Morrissey, John H. "Verifier for Handwritten Signatures", United States Patent 3,983,535, assigned to IBM Corporation, September 28, 1976 Cited in Chainer85b: for acceleration on signature verification Signature verification using acceleration: acceleration/accelerometer stylus (no table), compare with Hewitt Crane Stanford accelerometer pen [HerbstNM77a] Herbst, Noel Martin and Morrissey, John Henry "Segmentation Mechanism for Cursive Script Character Recognition Systems", United States Patent 4,024,500, May 17, 1977, assigned to IBM Corporation, Armonk, New York Cursive character segmentation using sharp inflection points (zero velocity) Also makes use of baseline and midline (top of lower-case characters) for character segmentation Criticizes "retrospective" character segmentation in script, such as Harmon [HerbstNM77b] Herbst, N.M., et al "Signature Verification Based on Complete Accelerometry", IBM Technical Disclosure Bulletin, Vol 19 No 12, May 1977, pp Stylus for digitizer, no tablet: uses acceleration (chain codes? compare to SRI pen) [HerbstNM78a] Herbst, N.M. and Liu, C.H. "Signature Verification Method and Apparatus Utilizing both Acceleration and Pressure Characteristics", United States Patent 4,128,829, December 5, 1978, assigned to IBM Corporation Armonk, New York Signature verification: force/pressure and acceleration. (BLRT chain codes): position features subject to most variability. Separate X and Y acceleration for signature verification. [HerbstNM79a] Herbst, N.M., Liu, C.N., and Panissidi, H.A. "Pressure sensing device and transducer arrangement", United States Patent 4,142,175, February 27, 1979, assigned to IBM Corporation Cited in Chainer85b acceleration on signature verification [HerbstNM79b] Herbst, Noel M., Liu, Chao N., and Panissidi, Hugo A. "Pressure Sensing Device and Transducer Arrangement", United States Patent 4,142,175, assigned to International Business Machines Corporation, Armonk, New York, February 27, 1979 Signature verification: pressure/force sensing stylus, not sensitive to lateral force angle: combined with acceleration for signature verification 8:26:16 PM]

60 ( ) History of Pen and Gesture Computing [HerotCF76a] Herot, Christopher F. "Graphical Input through Machine Recognition of Sketches", Computer Graphics, Vol 10 No 2, Summer 1976, pp , Proc. SIGGRAPH '76 Sketch recognition: segmentation of tablet input into lines and splines based on speed an pressure. Success of sketch recognition / drafting in part because system is interactive (graphical user interface) for which user may manipulate and modify recognition. Inking stylus. [HerotCF76b] Herot, Christopher F. "Sketch Recognition for Computer-Aided Design", Proc. URODIGS '76, ACM/SIGGRAPH Workshop on User-oriented Design of Interactive Graphics Systems, 1976, pp Sketch recognition: infer user intent (for segmentation, etc.) based on speed, sequence and pressure (tipforce) of strokes. E.g. if drawn more slowly, user probably intended those rounded corners, so do not sharpen them. Overtracings intended to lengthen mark, or make more bold. Tablet large format, has eraser sensor on rear end like a pencil (compare with Wang Freestyle tablet). [HerotCF78a] Herot, Christopher F. and Weinzapfel, Guy "One-Point Touch Input of Vector Information for Computer Displays", Proc. SIGGRAPH '78; SIGGRAPH Newsletter Vol 12 No 3, Aug 1978, pp Pressure-sensitive digitizer (also direction and torque-sensitive) 6DOF using roughened glass surface and finger as stylus, area for pressure/force. Pressure-sensitive display. Friction with finger not undesirable, can be used for multidimensional input. Can detect rotation of a single finger. Finger can be used to give initial push to graphical object, then steer without moving finger from position by detecting rotation/orientation of finger. Mentions two-finger twist/rotate gesture. Mentions multiple finger / multitouch touchscreen under development with Elographics. Mentions kinesthetic (haptic) input devices. Compare to Artificial Reality project? [HerotCF80a] Herot, Christopher F. "Spatial Management of Data", ACM Trans. on Database Systems, Vol 5 No 4, December 1980, pp Graphical Data Space (GDS) with world view of data in one display (using small icons), local hierarchy context in another display, detail record in another. Early system for semantic zooming. Prototype uses joystick not mouse or digitizer. [HerroldPA79a] Herrold, P.A. "Alphanumeric CRT Display Having A Plurality Of Display Positions", IBM Technical Disclosure Bulletin TDB 09-79, p , 01-Sep-1979 CRT display can be rotated 90 degrees (landscape vs. portrait), outline (frame/window borders) of display and of text is automatically reformatted/regenerated for orientation. [HilbrinkJO78a] Hilbrink, Johan O. "System for Optically Entering, Displaying and Decoding Handwritten Symbols", United States Patent 4,122,415, September 5, 1978 Lightpen and television monitor display, when used for handwriting input and replacing the handwriting input with a generated character [Himmel78] Himmel, David P. "Some Real-World Experiences with Handprinted Optical Character Recognition", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-8 No 4, April 1978, pp How much effort it takes to train people to handwriting constraints (OCR). Shows samples of constrained writing people were actually trained to. [HoffmanHS80a] Hoffman, H.S. "Exact Position Detection with Finger Point and Feedback", IBM Technical Disclosure Bulletin 22, 1536 (September 1979) capacitive/electrostatic touchscreen grid that can be integrated with a display: interpolation between grid lines to get fine position accuracy writing [Hollerbach78] Hollerbach, J.M. "A Study of Motor Control Through Analysis and Synthesis of Handwriting", PhD Thesis, Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science, August, Also described in Visible Language, Vol XIII No 3, 1979, pp Study of stroke directions based on anatomy of muscles. Points out elimination of clock-wise movement in fast writing. [Hosaka77] Hosaka, M. and Kimura, F. "An Interactive Geometrical Design System with Handwriting", Proceedings of the 6th IFIPS Congress, Toronto, August 1977, pp User-interface: handwritten sketches, sketch recognition, gesture input to do drawings Refers to measuring problems of tablet (and of "human behavior") (human factors / variability). Uses second-level fix-up on recognition if first pass is inconclusive. Uses chain codes of inflection points, rotations, straight segments, etc. [HoshinoY77a] Hoshino, Y. "Word recognition apparatus", United States Patent 4,010,445, March 1977 Cited in Suen79 Dictionary of words in character recognition, look for entry with best/closest match, not exact match. [Hubel79] Hubel, D.H. and Wiesel, T.N. "Brain Mechanisms of Vision", Scientific American, No 241, 1979, pp Martin,GL90b [IBM76a] IBM "Electronic Calculator Based on Character Recognition of Input from Stylus Acceleration Dynamics", IBM Technical Disclosure Bulletin, December 1976, pp :26:16 PM]

61 ( ) History of Pen and Gesture Computing Accelerometer writing stylus (no table), used for calculator input. Compare to Hewitt Crane Standfor accelerator pen. [IBM79] IBM "Object Identification and Local Highlighting on Displays", IBM Technical Disclosure Bulletin, Jan 1979, p 3284 Compare with object highlighting via proximity tablet data of "GUI-handles" in a GUI: mentions getting local highlighting of an object on a display as the stylus goes over it, and returning a ID/handle to the application instead of X/Y coordinates. Mentions equivalent functions from lightpen, tablet, joystick, or other positioning (position input) device. [IchikawaT77] Ichikawa, Tadao, Sakamura, Ken, and Aiso, Hideo "ARES - A memory, capable of associating stored information through relevancy estimation", Proceedings of 1977 National Computer Conference, pp Seven-direction chain codes/ segment lengths, with an additional approximate string matching function "Lee Distance" from error-correcting codes (in hardware) [IkedoT78] Ikedo, Tsuneo "Automatic Coordinate Determining Device", United States Patent 4,088,842, May 19, 1978 Electromagnetic digitizer tablet that can handle multiple styli (multi-touch), and a cordless/wireless probe or stylus: Can use multiple probes/styli on one device. [ImpedovoS76a] Impedovo, S., Marangelli, B., and Plantamura, V.L. "Real-Time Recognition of Handwritten Numerals", IEEE Trans. Systems, Man, and Cybernetics, Vol 6 No 2, February 1976, pp % to 90% recognition, numerals only (!) Gives examples of handwriting variations: stroke directions [InfoWorld80a] InfoWorld "Write to a Micro", InfoWorld Magazine, October 13, 1980, page 22 MicroPad product announcements: handwriting terminal with two line display, handwriting recognition on pre-printed paper forms. Compare with PenPad from Pencept and CIC HandWriter [IshiiM79a] Ishii, Mitsuo; Yamamoto, Masanari; Iwasaki, Michiko; and Shiraishi, Hiroshi "An experimental input system of hand-drawn logic circuit diagram for LSI CAD", DAC '79 Proc. 16th Design Automation Conference, 1979, pp OCR static recognition of hand-drawn CAD electronic designs/circuits. Requires very neat/constrained handwriting input: connecting lines drawn with rulers, component symbols drawn using a template. [Ito78] Ito, M.R. and Chui, T.L. "On-line Computer Recognition of Proposed Standard ANSI (USASI) Handprinted Characters", Pattern Recognition, Vol 10, 1978, pp Claims 98.3% accuracy on A non-training system (non-adaptive) handwriting recognition. It's not just tablet digitizer noise: unsteadiness in user's pen motion. Eight-type chain code classifications. Straight-line vs curvilinear stroke classes. Testing for performance used "intentional variability" in characters by authors: testing bias? Cites mis-recognition due to missing stroke segments (K-R, J-T). Digitizer error? Delay in recognition same as time for user to look at screen: not a user-interface/human-factors problem. Variability: Shows bizarre four-stroke "Z" shape form. Shows proposed ANSI hand-print standard for static OCR recognition including stroke directions, constrained writing styles. [JarvisJF76a] Jarvis, J.F. "Regular expressions as a feature selection language for pattern recognition", Proceedings of the 3rd International Joint Conference on Pattern Recognition, Coronado, California, November 1976, pp [JarvisJF76b] Jarvis, John F. "The line drawing editor, an Experiment in Computer Vision", Proc. SigGraph '76, July 14-16, 1976, Philadelphia, pp Drawing editor / sketch recognition, mark-up commands (edit commands / gestures) written on paper, optically scanned to change drawings (no tablet!). Gesture shapes include digits, X, connector points, carets, arrow tabs. [JarvisJF77a] Jarvis, J.F. "The line drawing editor: Schematic diagram editing using pattern recognition techniques", Computer Graphics Image Processing, Vol 6, 1977, p Drawing editor / sketch recognition [JonesAK78a] Jones, Anita K. and Liskov, Barbara H. "A Language Extension for Expressing Constraints on Data Access", CACM, Vol 21 No 5, May 1978, pp Constraint-definition programming language extensions for fine-grain defining access control/capabilities (rights) of objects -- what other data/objects it can access, what rights another object must have to access it. Control fine-grained to type of operation also. See ContentGuard Google. [Kanal77] Kanal, Laveen N. "Current Status, Problems and Prospects of Pattern Recognition", Systems, Man, and Cybernetics Review, Vol 6 No 4, August, 1977, pp 9-14 Overview of pattern recognition: many theories, few products: term covers too many divergent problem areas. Pattern recognition overview: well-constrained problems, sustained efforts lead to successful performance. Realistic performance: "Identifying animals in a zoo is much different from hunting them in the forest." Overview: spatial/temporal/external context, feature extraction (most publicly undocumented: not published), experiment/training design, clustering, no standard data bases [Kaplow76a] Kaplow, Roy and Molnar, Michael "A computer-terminal, hardware/software system with enhanced 8:26:16 PM]

62 ( ) History of Pen and Gesture Computing user input capabilities; the enhanced-input terminal system (EITS)", ACM SIGGRAPH, Vol 10 No 2, Summer 1976, pp Dynamically variable keyboard, using digitizer/touch-screen and simulated function keys/buttons: simulated devices, compare with simulated devices and multiple-finger gestures for Schumer, Fingerworks, Buxton. Refers to multiple touches as "chord" mode for simulated keyboard on touchscreen. User-interface where icons/keyboard labels on touch-screen force/pressure-sensitive digitizer/touch-screen change dynamically. Gesture input as chorded/multi-button/multi-touch input. Application Generator? Soft buttons? [Kaplow80] Kaplow, R. and Molnar, M.K. "Dynamically variable keyboard system", United States Patent 4,202,041, May 6, 1980 assigned to Massachusetts Institute of Technology, Cambridge, Massachusetts, 1980 Dynamically variable keyboard, using digitizer/touch-screen and simulated function keys/buttons: simulated devices, compare with simulated devices for Schumer, Fingerworks, Buxton. User-interface where icons/keyboard labels on touch-screen force/pressure-sensitive digitizer/touch-screen change dynamically. (In chord mode at least) Keys are not selected until release, permitting user to adjust keys (position) before selecting. Supports make-before-break, rolling contact (N-key rollover), and purposeful multiple key contact. Gesture input as chorded/multi-button/multi-touch input? Application Generator? Soft buttons? Note: had multi-touch sensing. [KashiroY80a] Kashiro, Yoshikazu; Matsugasako; Kenji; and Kataoka, Shunro "Process for Producing an anisotropically Electroconductive Sheet", United States Patent 4,209,481, June 24, 1980 Anisotropic conductive (rubber) sheet: fine wires aligned by strong magnetic field during molding to reach from one side to the other. [Kato80] Kato, O, Fujita, T., Niwa, M., Morishita, T., Fujii, N., Tanahashi, J. "Handwriting Input System for Japanese", Proceedings of IFIP 80, Tokyo, Japan, October 6-9, 1980, North-Holland publishers, pp Science Citation Index. Features are stroke-relative position, then candidate selection, then personal training. Claims 98% recognition rate performance on Japanese Kanji/kana/numerics, after training. [KayAC76a] Xerox Palo Alto Research Center, Learning Research Group (Kay, A.C.) "Personal Dynamic Media", Xerox PARC Learning Research Group, 1976 Dynabook, original report. Musical score capture system OPUS using music keyboard. Turtle geometry with command sequences / scripts. One mention of touch-sensitive screen / touchscreen for future projects, not part of physical Dynabook. [KayAC77a] Kay, A.C. "Microelectronics and the Personal Computer: Personal Dynamic Media", Scientific American, September 1977, page 231 The original user-interface paper: Dynabook (does NOT mention electronic ink or character recognition). See also longer Xerox report. [KayAC77b] Kay, A.C. and Goldberg, A. "Personal Dynamic Media", IEEE Computer, Vol 10 No 3, March 1977 pp Dynabook did NOT have stylus or handwriting input: mouse only.. popularly assumed to be the original "electronic ink" handwriting paper: Dynabook, but not so. See also longer Xerox report same title. [Kegel78] Kegel, A.G., Giles, J.K., and Ruder, A.H. "Observations on Selected Application of Optical Character Readers for Constrained Numeric Handprint", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-8 No 4, April 1978, pp Most handwriting recognition OCR problems are over-printing, 0-vs-6 ambiguity. Also how well human writers match constraints for various commercial OCR handwriting recognition products [KentST76a] Kent, Stephen T. "Encryption-Based Protection Protocols for Interactive User-Computer Communication", MIT Lab. for Computer Scients Technical Report MIT/LCS/TR-162, May 1976 Encrypted communications/networking in Multics: chain based block cipher [KickertWJM76a] Kickert, Walter J.M. and Koppelaar, Henk "Application of Fuzzy Set Theory to Syntactic Pattern Recognition of Handwritten Capitals", IEEE Trans. Systems, Man, and Cybernetics, Vol 6 No 2, February 1976, pp 148-ff [KleyV78a] Kley, Victor "Graphical data entry pad", United States Patent 4,079,194, March 14, 1978 Resistive sheet touch tablet, using two resistive layers: bottom layer top layer scanned separately for X and Y. [KnowltonKC76a] Knowlton, Kenneth Charles "Method and apparatus for using pushbutton telephone keys for generation of alpha-numeric information", United States Patent 3,967,273, June 29, 1976 Use telephone keypad (3 high x 4 wide) with multiple QWERTY alphabetic characters on keys. Use hits a key to select a group, then keys to select position in group, to get a particular character. [KnowltonKC77a] Knowlton, K.C. "Computer Display Optically Superimposed on Input Devices", Bell System Technical Journal, Vol 56 No 3, March 1977, pp User-interface: project image on keyboard to change keys: refers to tablet input possibility; compare with Kaplow for soft buttons. Mentions using X-Y tablet (digitizer) with display. [KnowltonKC80a] Knowlton, Ken "Progressive Transmission of Grey-Scale and Binary Pictures by Simple, Efficient, 8:26:16 PM]

63 ( ) History of Pen and Gesture Computing and Lossless Encoding Schemes", Proceedings of the IEEE, Vol 68 No 7, July 1980, pp Compression of images by run-length and difference encoding: lossless [KohnfelderLM78a] Kohnfelder, Loren M. "Towards a Practical Public-key Cryptosystem", Undergraduate thesis, M.I.T., May, 1978 Original source for concept of "digital certificate" for PKI infrastructure, trusted authority certifies identity to go with a particular public/private key pair. Discusses Merkle, Merkle-Hellman, and RSA public-key algorithms. Certificate is alternative to Public File / contacting central certifying directory on each use. Public File == CA certification authority. [KruegerMW77a] Krueger, Myron W. "Responsive Environments", Proc. NCC 1977, pp VIDEOPLACE -- optical/video sensing of user outlines in a room-sized environment: non-realistic interactions possible e.g. two users' images must combine to create a window into another scene, etc. Multiple-user, compare optical sensing with multi-touch sensing, interactive desk environment. [KuipersJ76a] Kuipers, Jack "Tracking and Determining Orientation of Object Using Coordinate Transformation Means, System and Process", United States Patent 3,983,474, assigned to Polhemus Navigation Sciences, Incorporated, Burlington, Vermont, September 28, 1976 From a old list of Polhemus patents Patent on correction tables on stylus angle, orientation, position in a 3- D tablet digitizer [KuipersJ77a] Kuipers, Jack "Apparatus for generating a nutating electromagnetic field", United States Patent 4,017,858, assigned to Polhemus Navigation Sciences, April 12, 1977 Cited in MaedaK87 three-dimensional / six dimensional electromagnetic digitizer, using X/Y/Z coils. Determines both position and orientation of stylus, thus able to report not only the position on a (threedimensional) surface, but also the plane of the surface at that point. [KuklinskiT77a] Kuklinski, T. "Implementing a Functional Attribute Based Theory of Character Recognition", unpublished manuscript, Pencept, Incorporated, March 21, 1977 Kuklinski voting algorithm for an imperfect decision tree based on pair-wise voting [KuklinskiT79a] Kuklinski, T., PhD Thesis: "Graphical Context as an Aid to Character Recognition", Massachusetts Institute of Technology, Cambridge, Massachusetts, 1979 Graphical context in character recognition: if all the characters are squished down low (eg. K, Y, E, H, etc.), then the mount of low-tail needed to distinguish a pair of similar characters (e.g. V versus Y) is also less. Really gets into similarity of distinguishing features as setting thresholds. [Kurzweil80a] Kurzweil Computer Products "Kurzweil Data Entry Machine", Kurzweil Computer Products, 1980 Kurzweil OCR product: Omni-Font Character Recognition, describes multiple algorithms for recognition, apparently to speed up processing. Includes print-to-braille, special modules for scanning telephone books to detect fields. [Kwan79] Kwan, C.C., Pang, L., and Suen, C.Y. "A Comparative Study of some Recognition Algorithms in Character Recognition", Proceedings of International Conference on Cybernetics and Society, pp , October 1979, IEEE order number CH1424-1/79/ Boundless variability of hand-print handwriting [Lambden78] Lambden, Martin Roy "Electrographic Apparatus and HandRest for Use Therewith", United States Patent 4,129,746 assigned to Quest Automation Limited, Stapehill, Wimborne, England, December 12, Hand rest for pressure-sensitive / resistive-film tablet, so that hand does not press on tablet surface [Lambden78b] Lambden, Martin Roy "Electrographic Apparatus and Method Of Producing and Electrode Surface Therefor", United States Patent 4,070,544 assigned to Quest Automation Limited, Stapehill, Wimborne, England, January 24, 1978 Description of the resistive-film, plastic pressure sheet digitizer tablet used for the Quest handwriting product [Larsen77] Larsen "Quadrant Check for Signature Verification", IBM Technical Disclosure Bulletin, Vol 20 No 4, September 1977, pp Accelerometer pen/stylus design (like SRI pen), produces chain codes [Lerner80] Lerner "Understanding speech proves tough task for machines", High Technology, April 1980 Magazine article [Lew79] Lew, J.S. "Optimal Designs of Instrumented Pens for Signature Verification", IBM Technical Disclosure Bulletin, Vol 21 No 8, p. 3415, January 1979 Variation on SRI (non-digitizer) pen: accelerometer pen stylus design [Liljenwall79] Liljenwall, James and Moss, David "Information Entry System", United States Patent 4,139,837, assigned to Creative Ventures, Incorporated, Dayton, Ohio, 1979 On-line character recognition on the face of a wristwatch or calculator: unistroke/single-stroke characters, written with a fingertip Simple on-line character recognition using zones/regions recognition [Lin78] Lin, W.C. and Pun, J.H. "Machine Recognition and Plotting of Hand-Sketched Line Figures", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-8 No 1, January 1978, pp :26:16 PM]

64 ( ) History of Pen and Gesture Computing Symbol recognition and graphical editing: "prettyizing" (prettifying) a sketch : shows recognition of singlestroke circuit-diagram symbols "recognition -- seems to be a problem at times since the user cannot have complete control over the input data when sketching on the tablet" [Lin79] Lin, C.N., Herbst, N.M., and Anthony, N.J. "Automatic signature verification: System description and field test results", IEEE Trans. Systems, Man, and Cybernetics, Vol 9, 1979, pp Cited in Teulings86a [LiuCN79] Liu, C.N., Herbst, N.M., and Anthony, N.J. "Automatic Signature Verification: System Description and Field Test Results", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-9 No 1, January 1979, pp Signature verification system from IBM using force/pressure and accelerometry Signatures: reference signature modified before storage to deal with variability Signature verification: 1.7% Type I (false rejection) and 0.4% Type II (false acceptance) for deliberate (expert) forgers, false acceptance rate of 0.02% for random (amateur) forgers Signature verification: 1% of users had too much variability for the system to work well: solution is administrative Signature hardware problems: baseline drift in force/pressure sensor, noise in force/pressure sensor, lack of reset, non-uniformity of ink cartridges [MIT76a] Massachusetts Institute of Technology "MIT Reports on Research: Reading Writing", MIT Reports on Research, Vol 4 no 3, November 1976, pp Press report on Blesser/Shillman work on handwriting recognition, predecessor to PenVerter Partners / PenCept. Examples are U-V and A-H functional attributes. Use case / application is replacing keypunch operations. [MackworthAK77a] Mackworth, Alan K. "On reading sketch maps", Proc. IJCAI '77, 5th Intl'. Joint Conf. on Art. Intelligence, Vol 2 pp MAPFREE sketch recognition and sketch editing using graphical data tablet. Segmentation of sketches into chains (of marks) and regions (of area), syntactic rules for joins of chains (junctures) [MadeiraM78a] Madeira, Marion "Gesture Recognition and the Use of Touch Sensitive Color Displays for Simple Diagramming", Master's Thesis, M.I.T, August 11, 1978 GESTUR gesture recognition on touch-sensitive device digitizer, using adaptive/trainable recognizer by Heckbert. Drawing program for entering diagrams/sketches during typing. Rubout gesture for erase, "L" angle gesture for rectangle. [MatsushitaT78] Matsushita, T. et al "Personal Computer-Aided Composition and Editing System", 3rd USA-Japan Computer Conference, 1978, 10/10-12/78 cited for text-editing of handwritten marks, graphical editing [MattesHG76a] Mattes, Hans George and Torok, Gabor Peter "Graphic Input Terminal", United States Patent 3,959,585, May 25, 1976 Resistive sheet touch panel using two sheets with voltages applied in X and Y directions on the two sheets, with an unenergized third sensor sheet between. [Maugh78] Maugh, Thomas H. II "Holographic Filing: An Industry on the Verge of Birth", Science, Vol 201, August 4, 1978, pp Cited in Sziklai84 on storing optical images, with signatures. Holographic storage would be more reliable, as losing part of holographic image leaves full image recoverable, albeit at lesser signal/noise ration. Analog storage, not digital. [McDermott80] McDermott, J. "R1: An Expert in the Computer Systems Domain", Proceedings of the First Annual National Conference on Artificial Intelligence, 1980, pp Expert A.I. system to configure orders for DEC VA computers [MerkleRC80a] Merkle, Ralph C. "Protocols for Public Key Cryptosystems", Proc IEEE Symp. on Security and Privacy, pp Review of weaknesses of key distribution / key management for public-key cryptography. Vulnerability of Kohnfelder certificates is compromise of central keys. [Mihelic77] Mihelic, F.; Pavesic, Nokola and Gyergyek, Ludvik "Recognition of Writer of Handwritten Texts", Proceedings of the International Conference on Crime Countermeasures, Kentucky University, Lexington, 1977, pp Cited by Marlin Eller, Microsoft Pen Computing group [Milloy78] Milloy, D.G. "Comment on recognition and confusion of the lower-case alphabet", Perception and Psychophysics, 1978, Vol 24 No 2, pp Substitution errors in human recognition of lower-case characters Criticizes defects in everybody's studies of lower-case character confusion matrices for human recognition [Moayer76] Moayer, B. and Fu, K.S. "A tree system approach for fingerprint pattern recognition", IEEE Trans. Computers, Vol C-25, 1976, pp [MoogRA77a] Moog, Robert A. "The Human Finger -- A Versatile Electronic Musical Instrument Component", Proc. 58th Conv. of Audio Engineering Society AES, November 4..7, 1977, New York Touch-sensitive input for electronic musical instruments: capacitive sensing found in Theremin. 8:26:16 PM]

65 ( ) History of Pen and Gesture Computing Touchpad/tablet sensing finger two-dimensional position (single finger) on array of capacitive pads driven by two frequencies, phase used to determine position. [MorenoR78a] Moreno, Roland "Systems for Storing and Transferring Data", United States Patent 4,092,524, May 30, 1978 Chip-card/chip-karte, defined as electronic object. Combine shorter password (i.e. PIN) with cryptographic chip-card to get security equivalent to longer password. [MoriS80] Mori, S. and Suen, C.Y. "A Study of the Zero and Letter Oh Problem", Proceedings of 5th International Conference on Pattern Recognition", December 1980, pp Cited in Litvin82 [Morrissey76] Morrissey, J.H. "Electronic Calculator Based on Character Recognition of Input from Stylus Acceleration Dynamics", IBM Technical Disclosure Bulletin, Vol 19 No 7, December 1976, pp Disclosure of US patent 3,983,535 for signature verification, but says extension to input for handwriting recognition is obvious. On user-interface of writing on an electronic calculator (using chaincodes/acceleration dynamics) [MullerAM77a] Muller, Alex Marcel and Tyrrell, Joseph Leslie "Electrical Sensing Writing Pen", United States Patent 4,034,155, July 5, 1977 Pen for electrostatic/capacitive grid tablet [Muth77] Muth, F.E. and Tharp, A.L. "Correcting human error in alphanumeric terminal input", Information Processing Management, Vol 13, 1977, pp Cited in Srihari83: on context/spelling correction [MyAppleSpace79a] MyAppleSpace.com "Apple Graphics Tablet 1979 Summagraphics Magnetostrictive", MyAppleSpace.com, fetched 2012 Summagraphics magnetostrictive tablet for Apple computers, Filed under MyAppleSpace11a. [Myers80] Myers, C.S., Rabiner, Lawrence R., and Rosenberg, A.E. "Performance tradeoffs in dynamic time warping algorithms for isolated word recognition", IEEE Trans. Acoustics, Speech and Signal Processing, Vol ASSP- 28, 1980, pp Cited in Kruskal83 [Narasimhan80] Narasimhan, M.A., Devarajan, Venkat, and Rao, K.R. "Simulation of Alphanumeric Machine Print Recognition", IEEE Trans. on Systems, Man, and Cybernetics, Vol SMC-10 No 5, May 1980, pp Shows performance results in a confusion matrix for alphabet from OCR data, compare with Shillman, who uses a confusion matrix to show which features are distinguishing [Naus76] Naus, M.J., and Shillman, R.J. "Why a Y Is Not a V: A New Look at the Distinctive Features of Letters", Journal of Experimental Psychology: Human Perception and Performance, Vol 2 No 3, 1986, pp Shillman article on functional attributes [NeedhamRM78a] Needham, Roger M. and Schroeder, Michael D. "Using encryption for authentication in large networks of computers", CACM Vol 21 no 12 December 1978, pp PKI authentication techniques in communications: pre-dates SSL, certificates. Authentication protocols (exchange of keys), signed communication, off-line authentication. Presumes secure computing platform (trusted system). Database of public keys/trusted authentication authorities instead of certificates, but uses hierarchical naming of trusted authorities. [NewScientist79a] New Scientist "Computer becomes a secretary: Quest Automation Micropad", New Scientist, 18 October 1979, page 190 Quest Automation Micropad (based on earlier Datapad) handwriting recognition terminal. Based on work from National Physical Laboratory in Teddington. Mentions that recognition uses stroke direction also, e.g. to distinguish sloppy 8 from sloppy B. Advertising plays on keyboard reluctance of managers compared to secretarial staff. [Niemann77] Niemann, Heinrich "Classification of Characters by Man and Machine", Pattern Recognition, Pergamon Press 1977, Vol 9, pp Human performance on OCR recognition of handwritten characters compared with machine performance, under various noise conditions: machine works better on typed fonts, human on handwritten characters [OBrien79] O'Brien, M.T. "A Network Graphical Conferencing System", Rand Corporation, Stanta Monica, California, August 1979 (N-1250-DARPA) Real-time freestyle user interface for electronic ink: a central blackboard/whiteboard conferencing / drawing communally, separate windows for each user for typing (words). [Odaka79] Okada, Jazumi, and Masua, Isao "Some Results on Stroke Order Free On-line Character Recognition", PRL Vol 79 No 29, pp (in Japanese) stroke-order independence by grouping strokes which are close together (?) [Odaka80] Okada, Jazumi, and Masua, Isao "Online Recognition of Handwritten Characters by Approximating Each Stroke with Several Points", Trans. IECE, 1980, Vol 63-D, No 2, pp (in Japanese) See same-title paper in IEEE SMC, Nov/Dec :26:16 PM]

66 ( ) History of Pen and Gesture Computing [OkaM78] Oka, Masatomo and Yasuhara, Makoto "Signature Verification Experiment based on Nonlinear Time Alignment: A Feasibility Study", IEEE Trans on Systems, Man and Cybernetics, Vol SMC-7, No 3, pp Dynamic time-warping / non-linear time alignment via dynamic programming for signature verification [Okuda76] Okuda, T., Tanaka, E. and Kasai, T. "A method for the correction of garbled words based on the Levenshtein metric", IEEE Trans. Computers, Vol C-25, February, 1976, pp DosterW77 on spelling dictionary? [Pallishusky76] Pallishusky, Silvia Minka "Method and Means for Teaching Alphabet Recognition and Learning Handwriting", United States Patent 3,950,863, April 20, 1976 Overlaying transparencies to teach children the order of strokes for learning handwriting Refers to nine basic shapes, or segments, from which all 26 letters (52 with lower case) are constructed [PapertS80a] Papert, Seymour "MINDSTORMS: Children, Computers, and Powerful Ideas", Basic Books, 1980 Primarily about Logo language and "turtle" programming for teaching children in exploring mathematical concepts, etc. Mentions speculative idea of a sketchpad computer where users could draw with programming, or by pointing with a "wand". Cited for concept of pen or touchscreen computer. [Pavlidis77a] Pavlidis, Theo "Structural Pattern Recognition", Springer-Verlag, New York, 1977 Defines pattern recognition as "identification of the ideal which a given object is made after" Definition: classification theory vs boundary theory [Pavlidis77b] Pavlidis, Theo "Comments on Current Perspectives in Pattern Recognition", Systems, Man, and Cybernetics Review, Vol 6 No 4, August, 1977, pp 8-9 Just because humans have good pattern recognition performance, and poor arithmetic, does not mean pattern recognition is easier than computer arithmetic [Pavlidis77c] Pavlidis, Theo "Polygonal approximations by Newton's method", IEEE Trans. Computers, Vol C-26, 1977, pp [Pavlidis80] Pavlidis, Theo "Algorithms for shape analysis and waveforms", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PARM-2, July 1980, pp Chain codes, corner detection, curvature, Fourier shape, shape analysis for OCR image recognition (segmented pictures) [Pencept80a] Pencept, Inc. "Pencept: New Concepts in Source Data Capture", Pencept, Inc (hardcopy book) Pencept business plan for financing, handwriting recognition tablet. Cites figures on productivity gain (faster) for data capture with handwriting than with typing/keyboarding. [PenzPA80a] Penz, Perry A. "Method of using a liquid crystal display device as a data input device", United States Patent 4,224,615, September 23, 1980 Use segments of LCD display (example is character displays) as capacitive sensors in touchscreen: deflection of a micron or so is enough to detect change in capacitance. [PepperW78a] Pepper, William Jr. "Human-Machine Interface Apparatus", United States Patent 4,071,691, January 31, 1978, assigned to Peptek, Incorporated, Bethesda, Maryland Phase-detection digitizer, signals with different frequencies applied to opposite sides, resistive film touchscreen responds to finger touch: compare to Scriptel? [PepperW78b] Pepper, William Jr. "Human-Machine Interface Apparatus", United States Patent 4,129,747, December 12, 1978 Cited by Sklarew88b on user interfaces on touchscreens. touch-screen digitizer using phase delay of a signal in a resistive film with capacitive coupling [PepperW80] Pepper, William Jr. "System for producing electric field with predetermined characteristics and edge terminations for resistance planes therefor", United States Patent 4,198,539, April 15, 1980 Resistive-film digitizer using non-linear and three-dimensional electric fields, two voltage sources. Fields can be curved as desired, or corrected for pin-cushion distortion. Improvement to 4,071,691: capacitive digitizer, cited in Microtouch patent as sensor for pen or finger controller [Persoon77] Persoon, E. and Fu, K.S. "Shape discrimination using Fourier descriptors", IEEE Trans. Systems, Man and Cybernetics, Vol 7, 1977, pp Taxt90: Pavlidis80 says 95% on IEEE handwriting recognition database [PetersonJL80a] Peterson, J.L. "Computer Programs for Detecting and Correcting Spelling Errors", CACM, Vol 23 no 12, December 1980, pp Cited in Sinha88: on spelling context correction Has many reference on spelling correction, detection, bestmatch to strings, limits of lack of syntactic and/or semantic context [Price79] Price, K. and Reddy, R. "Matching segments of images", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-1, January, 1979, pp Cited in Burr81 [PrughRW80a] Prugh, R.W. and Fadden, B.J. "Graphic Digitizer", United States Patent 4,206,314, June 3, 1980, assigned to GTCO Corporation, Rockville, Maryland 8:26:16 PM]

67 ( ) History of Pen and Gesture Computing Digitizer with grid as transmitter, loop as the receiver [PurcellSC77a] Purcell, Stephen C. "Understanding hand-printed Algebra for computer tutoring", Massachusetts Institute of Technology, Artificial Intelligence Laboratory Memo No 445, February 1977 (also available as S.M. Thesis) 2D programming language with tablet and symbol/handwriting recognition user interface. Uses domainspecific knowledge (i.e. mathematical expressions) / context in parsing and recognizing mathematical expressions with on-line handwriting recognition. "chalkboard languages" - two-dimensional doodles, writing, pictures Points out that overlap is inadequate as the only criterion for parsing/segmentation in handwriting (two-dimensional) Uses a lattice representation of possible stroke groupings/segmentations of characters, showing ambiguities of parsing (vs. of recognition) Features: local geometry (curvature, etc.) not enough: need global geometry (closure of endpoints on "U" vs. "O", etc.) Variability: greater when humanswritein context, vs one character at a time Recognizer mentioned by Alan Kay in posting on November 2000 [Rabiner78] Rabiner, Lawrence R., Rosenberg, A.E., and Levinson, S.E. "Considerations in dynamic time warping for discrete word recognition", IEEE Trans. Acoustics, Speech, and Signal Processing, Vol ASSP-26, 1978, pp Cited in Kruskal83 [Rabiner79] Rabiner, Lawrence R., and Wilpon, Jay G. "Speaker-Independent Isolated Word Recognition for a Moderate Size (54 Word) Vocabulary", IEEE Trans. Acoustics, Speech, and Signal Processing, Vol ASSP-27 No 6, pp Discrete word speech recognizer using statistical clustering templates: a carefully-trained speakerindependent limited recognizer can work as well as casually-trained speaker-dependent Speech recognition on a professionally trained actor with very human-clear diction was worse Clustering procedure for voice templates much better than casual enrollment Describes many enrollment/training methods for speakerindependent/dependent recognition system Training/enrollment data was "cleaned up" to remove breathing sounds, clicks, etc. [Rabiner80] Rabiner, Lawrence R. and Schmidt, C.E. "Application of dynamic time warping to connected digit recognition", IEEE Trans. Acoustics, Speech, and Signal Processing, Vol ASSP-28, 1980, pp Voice recognition. Dynamic time warping to locate images of digits in a string: gives probabilistic results for full string, depends on post-processing to select final result. Segmentation not known in advance, nor how many digits in the utterance. Cited in Kruskal83 [RadicePF80a] Radice, P.F. "Personal Verification Device", United States Patent 4,234,868, November 18, 1980, assigned to Penwalt Corporation, Philadelphia, Pennsylvania Signature verification using force/pressure biometric only. "force/pressure only" digitizing tablet using piezoelectric or pyro-electric sheet or film [RahuelJC80] Rahul, Jean-Claude and Dagnelie, Jean-Paul "Graphics table particularly for a telewriting system", United States Patent 4,225,750, September 30, 1980 X/Y electromagnetic grid tablet digitizer operating at 10 Megahertz, modulated at 100 Kilohertz. Cites application as a telewriting (telautograph?) device. Cites to 1973 references for electronic whiteboard/blackboard systems. [RaoK80a] Rao, Kameswara and Balck, Kenneth "Type Classification of Fingerprints: A Syntactic Approach", IEEE Trans. PAMI, vol PAMI-2 no 3, May 1980, pp Classification of fingerprints using syntactic chain codes / BLRT patterns: loop, whorl, double-whorl, etc. [Reenskaug79] Reenskaug, Trygve "A note on Dynabook requirements", Xerox PARC memo, 22 March 1979: partial copy available on the Internet Automatic layout / structuring of graphical objects with connections: note that it does not mention handwriting recognition or character recognition? Refers to Prokon Anticipates market for Dynabook-like products about , programming in a higher-level language (e.g. APL, Pascal) [ReevesW78a] Reeves, William; Buxton, William; Pike, Robert; and Baecker, Ron "Ludwig: an Example of Interactive Computer Graphics in a Score Editor", Structured Sound Synthesis Project (SSSP), U. Toronto, Canada, November 1, 1978 Graphical editor for music scores, using digital tablet. Has a drop-down (see-through/transparent?) tool, vertical menu of note durations, user drags it to put a particular duration over a note and release: transparent note bar/menu? [RetzDL76a] Retz, David L. and Schafer, Bruce W. "Structure of the ELF operating system", Proc. NCC National Computer Conference, 1976, pp Multi-user operating system for DEC PDP-11 computers, used in Arpanet development. USERLOADER utility to load (download) programs from another system on the Arpanet network, and install (map into virtual address space). Mentions TelNet, control-c EXEC character to interrupt programs. [RivestRL78a] Rivest, R.L.; Shamir, A.; and Adleman, L. "A Method for Obtaining Digital Signatures and Public- 8:26:16 PM]

68 ( ) History of Pen and Gesture Computing Key Cryptosystems", CACM, Jan 1983, Vol 26 No 1, pp Original RSA paper on PKI public key cryptography / public key cryptosystems. Example application is secure, signed . Does not mention CAs. [RosenfeldA78a] Rosenfeld, Azriel "Survey: Picture Processing: 1977", (source not known) 1978 Not sure of source: Check Kuklinski's thesis? Large bibliography on picture processing, filtering, edge detection, image compression, character recognition and other kinds of pattern recognition (fingerprint, face, etc.), machine vision [RosenfeldA80a] Rosenfeld, A. "Picture processing: 1980", Computer Graphics and Image Processing, Vol 16, May, 1981 Fu86/Fu83 [RubincamDP79a] Rubincam, David P. "Electronic Book", United States Patent 4,159,417, June 26, 1979 Cited in Sklarew88b Portable electronic book, with user interface (no computer applications): book text contained on memory card, shown in display. Hinged notebook-like design. Compare with Dynabook by Alan Kay? [SIGGRAPH79] SigGraph "General Methodology and the Proposed Core System", report of the Graphics Standards Planning Committee, in Computer Graphics (Proc. of SIGGRAPH 79), Vol 13 No 3, August 1979 [SMC78] (various authors) "Special section on hand-print OCR", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-8 No 4, pp 279 ff, April 1978 Special section on real-world application experience with hand-print OCR, vs laboratory results [Saich77a] Saich, William George "Data Terminals having Interactive Keyboards and Displays and Data Processing Apparatus Incorporation such Terminals", United States Patent 4,028,695, June 7, 1977 Soft button keyboard: touch of key causes change in appearance to indicate that it is selected, press (release?) of key causes function to be performed. Two gestures, one of which has a highlight? Or One gesture, with two kinds of feedback? [SakoeH76a] Sakoe, Hiroaki "Automatic Character Recognition Device Employing Dynamic Programming", United States Patent 3,979,722, September 7, 1976, assigned to Nippon Electric Co. Limited, Tokyo Japan Refers to "standard patterns" for template matching (which it is not). Eight-direction chain codes. Uses dynamic matching of vector maxima/minima for DCR handwriting recognition (BLRT chain codes). Chaincode BLRT points are absolute, vary distances between for degree of match. [SakoeH78a] Sakoe, Hiroaki and Chiba, S. "Dynamic programming algorithm optimization for spoken word recognition", IEEE Trans. Acoustics, Speech and Signal Processing, Vol ASSP-26, 1978, pp Cited in Kruskal83 dynamic programming method for speech recognition, discrete words: only categorized that error rate is one-third less, not overall accuracy or success. [SakoeH79a] Sakoe, Hiroaki "Two-Level DP-Matching -- A Dynamic Programming-Based Pattern Matching Algorithm for Connected Word Recognition", IEEE Trans. Acoustics, Speech and Signal Processing, Vol ASSP-27, NO 6, 1979, pp 588-ff dynamic programming method for speech recognition, continuous speech but with limited vocabulary: digits only, "sentences" of one to four digits, only five speakers in test data. [ScanData76a] Scan-Data Corporation "ScanData 2250/1 OCR System with handprint recognition", Scan-Data Corporation, 1976 OCR product for constrained handprint recognition, using very stylized block writing style. Recommended size, shape, style of handprint constraint boxes in designing forms, which must be in particular color separate from ink. [SchmandtC80a] Schmandt, Christopher "Some Applications of Three-Dimensional Input", Master's Thesis, M.I.T., January 1980 Study of using Polhemus three-dimensional digitizer for pointing and body position. The original work on "Put-That-There", combining speech recognition and hand pointing. [SchultzJ79a] Schultz, Jan and Davis, Layton "The Technology of PROMIS", Proc of IEEE, Vol 67 No 9, September 1979, pp PROMIS: Early Digiscribe SAW touch-screen user interface system for medical records, 1967, user interface mostly point-and-click plus keyboard for text input. [SeelbachHE80a] Seelbach, H.E. "Input Device for Input of Alphanumeric Characters into a Computer", United States Patent 4,184,147, January 15, 1980 Simple four-grid "character recognition" device: sequence of crossings [Sethi77] Sethi, I.K. and Chatterjee, B. "Machine recognition of constrained handprinted Devanagari", Pattern Recognition, Vol 9, pp 69-75, 1977 Cited in Biswas81 [Seuffert77] Seuffert, P. "An Application of Line and Character Recognition in Cartography", Proceedings of IEEE Conference on Pattern Recognition and Image Processing, pp , June 1977 Cited in Goodale83 8:26:16 PM]

69 ( ) History of Pen and Gesture Computing [ShillmanR76a] Shillman, R. "Automatic Recognition of Thick Stroke Characters", Quarterly Progress Report No 118, Appendix IX, Massachusetts Institute of Technology Research Laboratory of Electronics, July 1976 How "legs" appear shorter as they get thicker [ShillmanR76b] Shillman, R. and Naus, G. "The Distinctive Features of the Letters O and D", Quarterly Progress Report No 118, Appendix X, Massachusetts Institute of Technology Research Laboratory of Electronics, July 1976 Functional attributes in character recognition: distinguishing characters O vs D is (one-sided) symmetry. also "leg" attribute for Y an V, F and C, P and D, H and U, and A from O. [ShillmanR76c] Shillman, R., Kuklinski, T. and Blesser, B. "Psychophysical Techniques for Investigating the Distinctive Features of Letters", International Journal of Man-Machine Studies, Vol 8, pp , 1976 Classical letter shapes do not occur frequently in hand-printing Shillman article on functional attributes [ShillmanR77a] Shillman, R.J. and Babcock, R.T. "Preliminary Steps in the Design of Optical Character Recognition Algorithms", unpublished manuscript, Cognitive Information Processing Group, Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, 1977 Physical to functional rules (functional attributes) for 2 and Z at Massachusetts Institute of Technology (see Babcock) Example of 2-Z space, one space invading the other Good introductory example of physical to functional rules, how to pick the dominant feature for a character recognizer [ShillmanR78a] Shillman, Robert J. "The Application of Optical Character Recognition to Machine Translation", Manuscript draft, Massachusetts Institute of Technology, 1978 Manual re-typing of text (especially with different alphabets: Cyrillic, etc.) largest bottleneck on machine translation of printed documents. Review of OCR static optical character recognition for text conversion. [ShillmanR78b] Shillman, Robert J. "Character Recognition and the Data Entry Problem", Manuscript draft, Massachusetts Institute of Technology, 1978 Focuses on OCR static optical character recognition for data entry, versus DCR on-line dynamic character recognition. Manual re-typing of text (especially with different alphabets: Cyrillic, etc.) largest bottleneck on machine translation of printed documents. [ShinghalR79a] Shinghal, R. and Toussaint, G.T. "A bottom-up and top-down approach to using context in text recognition", International Journal of Man-Machine Studies, Vol 11, 1979, pp Cited in Sinha88: on spelling/dictionary context correction [ShinghalR79b] Shinghal, R. and Toussaint, G.T. "Experiments in text recognition with modified Viterbi algorithm", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-1, 1978, pp Cited in Sinha88: on spelling/dictionary context correction [Shortliffe76] Shortliffe, E.H. "Computer-Based Medical Consultation: MYCIN", American Elsevier, New York, 1976 Expert A.I. system to diagnose infectious blood diseases Cited in FoleyJD84 [Simmons80] Simmons, Robert M. "An On-line Character Recognizer", Interface Age, March 1980, pp Very simple on-line handwriting character recognition program running on a 1980 Z-80 CP/M system effectively trims ends of stroke by looking for matching chaincode sequence within longer chain-code sequences Zone-based recognizer, using chain of codes for the zones visited: single-stroke characters? [SimonJC77a] Simon, Jean Claude and Rosenfeld, Azriel "A short survey of character recognition", Digital Image Processing and Analysis, 1977 Overview of character recognition concepts: preprocessing, feature extraction, classification. Mostly literature reference list. [SipplCJ80a] Sippl, Charles J. and Sippl, Roger J. "Computer Dictionary, Third Edition", Howard Sams, 1980 (hardcopy book) Technical dictionary 1980 [SiyP78a] Siy, Pepe and Brown, Joseph R. "Signature Verification Pen", United States Patent 4,131,880, December 26, 1978 Stylus/pen for signature verification using tip force/pressure, piezoelectric force/pressure transducer/sensor [SmithAR78a] Smith, Alvy Ray "Color Gamut Transform Pairs", Proc. SIGGRAPH 78, Aug 1978, pp Reprint, "Tutorial: Computer Graphics" IEEE Computer Society, 2nd Ed., 1982, pp Early paper related to Alpha blending: mathematical treatment of color blending / teinting in digital graphics, in RGB space and HSV space. More than 8 bits per color imperceptible to human user eye. Hexcone model for color perception. Gamma correction / gamma compensation for nonlinearity of color scale. [SmithAR79a] Smith, Alvy Ray "Tint Fill", Proc. SIGGRAPH 79, 1989, pp ACM SIGGRAPH Computer Graphics Newsletter, Vol 13 No 2, August 1979 Area fill algorithm for shaded images, where boundaries are not sharp. Only changes tint of pixels, not brightness. [SnyderEJ77a] Snyder, Edward J. and Domyan, Stephen L. "Position coordinate determination device", United States Patent 4,018,989, assigned to Summagraphics Corporation, April 19, 1977 Cited in Lukis87 on digitizer tablets Ratiometric measurements in magnetostrictive grid tablet, to correct 8:26:16 PM]

70 ( ) History of Pen and Gesture Computing for errors that increase with tablet size. [Spanjersberg78] Spanjersberg, A.A. "Experiments with Automatic Input of Handwritten Numerical Data into a Large Administrative System", IEEE Trans. on Systems, Man, and Cybernetics, Vol SMC-8, No 4, April 1978, pp Shows various constraint methods on handwriting styles (boxes, zones, write around the dots, markedsense card) to make handwriting more recognizable by machine, gives some performance results [Spohrer80] Spohrer, James C., Brown, Peter F, Hochschild, Peter H., and Baker, James K. "Partial Traceback in Continuous Speech Recognition", IEEE publication /80/000/0036 Verbex Corporation, formerly Dialog Systems Inc, Two Oak Park, Bedford MA use of context in continuous speech recognition [Srihari76] Srihari, S. and Zack, G. "Document Image Analysis", Proceedings of 8th International Conference on Pattern Recognition, Paris, France, October 1986, pp Cited in Baird86a: on document skew alignment angle for OCR [Stallings76] Stallings, W. "Approaches to Chinese Character Recognition", Pattern Recognition, Vol 8, 1976, pp [StockholmE79a] Stockholm, E. "Recognition of a writer as a function of his method of writing", Perception and Motor Skills, Vol 49, 1979, pp (partial copy) Identify user by handwriting characteristics. Teulings86a cites for invariant spatial features in handwriting, regardless of writing conditions for subject (blackboard, paper, etc.) [StumpeB77a] Stumpe, Bent "A New Principle for an X-Y Touchscreen", CERN technical report, March Capacitive/electrostatic touchscreen digitizer using X and Y planes of conductive traces: earlier work goes back to Additional papers in file regarding conductive polymers for a transparent touchscreen. Senses on multiple traces and determines "gravity point" of touch location. Compare with Scriptel. [StumpeB78b] Stumpe, Bent "Experiments to find a manufacturing process for an x-y touch screen: Report on a visit to Polymer-Physik GmbH", CERN technical report, 6 February 1978 Manufacturing process for x-y touch screen based on StumpeB77a. Clear and insulating conductive lacquers to make conductive/resistive films, also with antireflective properties. Mentions putting lacquer between wires for better optical properties. [Suen76a] Suen, C.Y. and Radhakrishnan, T. "Recognition of hand-drawn flowcharts", Proc. 3rd Intl. Joint Conf. on Pattern Recognition, pp OCR Recognizing flowchart symbols: standardized shapes, strict syntax (chain codes) [Suen77a] Suen, C.Y. "Alphanumeric Hand-prints with Stroke Directions and Sequences", internal report for Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1977 Original collection of writing samples for PenVerter Partners (Pencept) [Suen77b] Suen, C.Y. and Shillman, R.J. "Low Error Rate Optical Character Recognition of Unconstrained Handprinted Letters Based on a Model of Human Perception", IEEE Trans. Systems, Man, and Cybernetics, Vol 7 No 6, June, 1977, pp Suen support (with Shillman) use of psychologically base features Suen claims the computer did better recognition than human on U-V [Suen78a] Suen, Ching Y. "Advances in Optical Character Recognition", Proc Canadian Computer Conf., Edmonton, May 23-25, 1978, pp. 1-6 Handprinting OCR recognition, uses templates [Suen79a] Suen, C.Y. "A Study on Man-Machine Interaction Problems in Character Recognition", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-9 No 11, pp , November 1979 Refers to "users do not follow models, but revert to usual writing" on training/constrained handwriting recognition Ergonomics / user-interface problems in handwriting recognition [Suen79b] Suen, C.Y. "n-gram statistics for natural language understanding and text processing", IEEE Trans. Pattern Analysis and Machine Intelligence", Vol PAMI-1, 1979, pp Cited in Sinha88: for statistics on language (one million word samples) to use for context correction Contains several lists of English-language word frequency dictionaries [Suen80a] Suen, C. "Computer Analysis of Handprinted Characters", Proceedings of the 5th International Conference on Pattern Recognition, Miami Beach, Florida, December, 1980, pp Human recognition success rate is 7.6% substitution error. Examples of handwriting variability from 1500 student enrollment cards, with distribution frequency Constraints: examples of how humans give cursive, lower-case, etc. even when asked specifically to print styles Handwriting recognition by machine with better performance than humans requires understanding of how humans recognize characters Taxonomy of different approaches to handwriting recognition: template matching, correlation; distribution of points, moments, crossings, distances; transformations and series expansions; geometrical and topological features, syntactic and structural analysis Describes Suen's research on distinctive features (functional attributes) Lists of constraints typically tried to reduce handwriting variability 8:26:16 PM]

71 ( ) History of Pen and Gesture Computing [Suen80b] Suen, C., Berthold, M. and Mori, S. "Automatic Recognition of Hand-printed Characters - The State of the Art", Proceedings of the IEEE, pp , Vol 68 No 4, April 1980 Survey of handwriting recognition, both static (OCR) and dynamic (on-line). Used as citation in several patents. Follow-up by Tappert and Suen in 1990, by Plamondon in Specifically mentions decision trees at several points. [Suen80c] Suen, Ching Y. "The Role of Multi-Directional Loci and Clustering in Reliable Recognition of Characters", Proc. 6th Intl. Conf. on Pattern Recognition, Munich, Oct 19-22, 1982, pp Fine angular resolution in OCR recognition of handwritten character by small features (characteristic loci). [Suenaga78] Suenaga, Yasuhito "Automatic text editing using handwritten mark recognition", Proceedings of the 9th Conference on Image Science and Engineering, No 3-1, pp 63-66, Dec (in Japanese) Cited in Suenaga83 [Suenaga80a] Suenega, Y. and Nagura, M. "A facsimile based manuscript layout and editing system by auxiliary mark recognition", Proceedings of 5th Int Conf on Pattern Recognition, Dec 1-4, 1980, pp Cited in Kankaanpaa87, Goodale83 See also Nagura83, same paper [Summagraphics78a] Summagraphics / Scriptographics "Summagaphics data tablet digitizer", Summagraphics, 1978 Electromagnetic tablet digitizer line. Shows large separate electronic controller: dual system HW-11/48 combines two tablets (large for digitizing, 11" x 11" small tablet for menus), one puck, on one controller and therefore half the data rate, with identifying bit/flag for tablet. Former name was Scriptographics. [Talmage78] Talmage, John E. and Bates, L. Dexter "Nonplanar Transparent Electrographic Sensor", United States Patent 4,071,689, January 31, 1978 Resistive-sheet digitizer using small spacing dots [TanLJ79a] Tan, Lu-Jan "Keyless electronic calculating pen", United States Patent 4,141,073, February 20, 1979 Accelerometer stylus/pen, recognition by zone-like directions of strokes (chain codes). Recognizes/evaluates mathematical/arithmetic expressions. Calculator display build into side of pen. Compare with Crane accelerometer pen, Anoto pen. [TappertCC78a] Tappert, C.C. and Das, Subratta K. "Memory and Time Improvements in a Dynamic Programming Algorithm for Matching Speech Patterns", IEEE Trans. Acoustics, Speech, and Signal Proc., Vol. ASSP-26, pp , 1978 Cited by Marlin Eller, Microsoft Pen Computing group [TappertCC78b] Tappert, C.C., and Kurtzberg, J.M. "Elastic Matching for Handwritten Symbol Recognition", Proc. IBM Int. Conf. Image Processing Pattern Recognition, Also, IBM Res. Rep. RC9988, May 1983 Cited by Marlin Eller, Microsoft Pen Computing group [TechTalk77a] Tech Talk "When is a U a V: Reading Computer Can Tell the Difference", M.I.T. Tech Talk, vol 21 no 28, March 9, 1977, p. 1 MIT newsletter: Blesser/Shillman and functional attribute (block) handwriting character recognition, later used by PenVerter Partners / Pencept. [Tersoff78] Tersoff, Abraham I. "Man-Machine Considerations in Automatic Handprint Recognition", IEEE Trans. on Systems, Man, and Cybernetics, Vol SMC-8 No 4, April 1978, p 279 short review: reliable OCR for hand-print is nowhere in sight [Teulings79] Teulings, H.H.M., and Thomassen, A.J.W.M. "Computer-Aided Analysis of Handwriting Movements", Visible Language, Vol XIII No 3, 1979, pp Describes digitizer set up to study handwriting: was tablet performance any good? [ThackerCP79a] Thacker, C.P.; McCreight, E.M.; Lampson, B.W.; Sproull, R.F.; and Boggs, D.R. "Alto: A Personal Computer", Xerox Corporation, Appeared in "Computer Structures: Principles and Examples", 2nd Edition, pp , Siewiorek, Bell and Newall, Eds. Original Alto personal computer with mouse and graphical user interface. Included BCPL language. Also mentions XY input tablet as input device, better than mouse for drawing, but tablet "interfered with keyboard" (no particulars given). [ThornburgDD79a] Thornburg, David D. "Capacitive Transducer", United States Patent 4,177,421 Capacitive / electrostatic digitizer (inventor of KoalaPad). Electrodes are tapered, to have greater capacitive coupling to finger depending on position. Alternate geometry has thing fingers along electrode at one end, thicker at the other. Compare with GTCO? [ThornburgDD80b] Thornburg, David D. "The PrestoDigitizer Tablet: A Low-Cost Device for the Real-Time On- Line Entry of Handprinted Characters", Proceeding of 3rd ACM SIGSMALL Symposium and the first SIGPC symposium on Small Systems, pp PrestoDigitizer product: seven-zone handwriting feature recognition. stroke sequence and direction information, simple dictionary to identify/recognize the handwritten character. Refers to learning algorithms (training of system?) [ThornburgDD80c] Thornburg, David D. "A Low-Cost Alternative to Data Entry Keyboards", Recreational 8:26:16 PM]

72 ( ) History of Pen and Gesture Computing Computing, January 1980, pp PrestoDigitizer product: seven-zone handwriting feature recognition [ThorntonRW79a] Thornton, R.W. "The Number Wheel: A Tablet Based Valuator for Three-dimensional Positioning", Proceedings of SIGGRAPH '79, published as Computer Graphics, Vol 13 No 2, August 1979, pp Essentially a one-directional trackball, two can be combined for simultaneous X and Y positioning (or angle) of 3D simulated object. Advantage over regular sliders is can do wrap-around (e.g for rotational angle), or implement hard limit stops at any value desired. Cited in FoleyJD84. Not 3D positional input. Implemented on a pen digitizer or touchscreen. [Toussaint77] Toussaint, G.T. "The Use of Context in Pattern Recognition", Proceedings of IEEE Computer Society Conference on Pattern Recognition and Image Processing, Renselear Polytechnic Institute, Troy, New York, June 6-8, 1977, pp 1-10 Context lies not just in the data, but in the perceiver Syntactical vs statistical use of context Statistical context: huge number crunching to reduce unmanageable probability distribution Types of word-feature context: graphological, phonological, statistical, syntactic, semantic, pragmatic Markov statistical methods for very large context dictionaries Modified Viterbi algorithm (statistical context) not so hot Goshtasby88 cites for using spelling dictionary look-up for context correction [Toussaint78a] Toussaint, G.T. "The use of context in pattern recognition", Pattern Recognition, Vol 10, 1978, pp Essay on fundamental need to consider context in pattern recognition: e.g. optical illusions in human perception; length of descenders on characters "P" and "D"; A vs. H, 8 vs. B; Roman "U" and "V" Cited in HullJJ83 [Toussaint78b] Toussaint, G.T. and Shinghal, R. "Cluster analysis of English Text", Proceedings of IEEE Conference on Pattern Recognition and Image Processing, Chicago, IL, 1978, pp Cited in HullJJ83 on context dictionary [TroxelDE76a] Troxel, D.E. "Feature Selection for Low Error Rate OCR", Pattern Recognition, Vol 8 No 2, April 1976, pp Scanned OCR character recognition (typed characters): need for strongly separating features, because of difficulty of dealing with long tails of probability distributions. Suggests placing character in confusion matrix to determine quality of classification by features used. [TsurutaS80a] Tsuruta, Shichiro; Sakoe, Hiroaki; Chiba, Seibi; and Nakada, Toshio "Connected Speech Recognition System DSP-100", NEC Research and Devleopment, No 56, January 1980, pp Speaker-independent continuous speed recognition system from Nippon Electric: dynamic programming, vocabulary words (Japanese), simple word matching then warping to match continuous speech. Tested on small number of subjects (7). [Tversky77] Tversky, A. "Features of similarity", Psychological Review, Vol 84, 1977, pp Cited in IchikawaS84: on human recognition [UllmannJR77a] Ullmann, J.R. "A binary n-gram technique for automatic correction of substitution, deletion, insertion and reversal error in words", Computer Journal, Vol 20, 1977, pp Spelling correction for transposed missing, and added characters in words. Correction of up to two errors by comparison to dictionary, special compression format for dictionary. Cited in Hull83a: on spelling correction [Viviani80] Viviani, P. and Terzuolo, V. "Space-time invariance in learned motor skills", in "Tutorials in motor behavior", Stelmach, G.E and Requin, J., editors, New York, North Holland, publishers, pp Cited in Schomaker86 Schomaker86 cites that overlearned/over-trained writing strokes (signatures?) show great invariance in relative stroke timing [Wakamatsu79] Wakamatsu, Shuichi "Character Writing System", United States Patent 4,144,405, March 13, 1979 Method for generating Kanji characters making use of the sequential order of writing parts of the characters with an input unit Shows tablet input of handwritten Kanji characters, but does not refer to character recognition per se [Wallace76] Wallace, Victor L. "The Semantics of Graphic Input Devices", "Proceedings of SIGGRAPH/SIGPLAN Conference on Graphics Languages, published as Computer Graphics, Vol 10 No 1, April 1976, pp Co-worker with Foley: Mentions virtual/simulated devices: relevant to Schumer, though not cited? Virtual devices (for semantic purposes) are Pick, Button, Keyboard, Locator, Valuator: specifically refers to character recognition for keyboard input: compare with Buxton virtual devices on a tablet. Early reference to logical equivalence of locator devices (tablet, touchscreen, mouse, joystick, etc.). Interactions/gestures include drag, rubber-banding, stroke (electronic "ink"). [Waltz78] Waltz, David L. "On the Interdependence of Language and Perception", U. Penn Technical Report T Partly an essay on the ambiguity of perception as a function of context 8:26:16 PM]

73 ( ) History of Pen and Gesture Computing [Ward76] Ward, J.R., Nelson, G.E., Desch, S.H., and Kaplow, R. "Two New Strategies for Computer-Assisted Language Instruction (CALI)", Foreign Language Annals, Vol 9 No 1, February 76, pp CAI Computer-aided instruction, grammatical/syntax analysis of German sentences: Gordon Eugene Nelson, Jean Renard Ward, Robert Kaplow (see Kaplow patent for UI item) [WhetstoneA76a] Whetstone, Albert L. and Phillips, Stanley C. "Digitizing_graphic_system_using_magnetostrictive_transducers", United States Patent 3,956,588, May 11, 1976 Summagraphics magnetostrictive tablet technology on drafting table, two magnetostrictive wires one each along each arm of mechanical X/Y drawing holder [WhiteGM78] White, G.M. "Dynamic programming for the Viterbi algorithm, and low cost speech recognition (abstract only)", Proceedings of the 1978 IEEE International Conference on Acoustics, Speech, and Signal Processing, pp Cited in Kruskal83 [WhiteJE76a] White, James E. "A High-Level Framework for Network-Based Resource Sharing", IETF RFC707: Early paper on making RPCs easier to use by standardizing message mechanisms for remote procedure calls: uses term Procedure Call Protocol. Separate error return message (e.g. exception). Marshalling is done by standardized parameter types: character strings, bit strings, integers, lists Cited in Kruskal83 [Wing78] Wing, A.M. "Response time in handwriting", in "Information processing in motor control and learning", New York: Academic Press, G.E. Stelmach, editor, 1978, pp Cited in Teulings86 as that up/down stroke pairs may be basic grapheme unit of writing [Wing79] Wing, A.M. "Variability in Handwritten Characters", Visible Language, Vol XIII No 3, 1979, pp Confuseable forms of script b-f, i-r Printing slower than script, but gets faster with practice People have same writing style on blackboard and on paper [WrightSJ78a] Wright, Jr., S.J., Anderson, P.T., and Grimes, R.S. "Multi-Modal Data Input/Output Apparatus and Method Compatible with Bio-Engineering Requirements", United States Patent 4,070,649, January 24, 1978 Accellerometer stylus, Character recognition using a four segment panel, diagonally. Reference similar to BLRT chain codes. Used for wrist-worn portable calculator with separate (wireless) accelerometer stylus. [Xerox76a] Xerox Corporation "Alto User's Handbook 15a", Xerox Corporation, 1976 User manual for Xerox Alto personal computer: three-button mouse, refers to operations when "hold it down" on a mouse button, prior to moving: press-and-hold? [Xerox78a] Xerox Corporation "The Alto User's Primer", Xerox Corporation, June 1978 Overview of operational procedures in Alto operating system: backups, initializing disk volumes, etc. [Xerox79a] Xerox Corporation "Alto User's Handbook September 1979", Xerox Corporation, September 1979 User manual for Xerox Alto personal computer. Sections Alto Non-programmer's Guide; Bravo Manual - text editor; Laurel Manual - electronic messages; Markup User's Manual, Draw manual - illustrations and drawing; FTP manual; Neptune manual - file management. Printing support. [Xerox80a] Xerox Corporation "Xerox 5700 Electronic Printing System Reference Manual", Xerox Corporation, 1980 Touch-panel / touch-screen interface for commercial printing system Xerox Virtual buttons: function of button depends on sequence of user input e.g. tapping on "help" button first changes function of next button user presses. Buttons displayed change depending on function. [Xerox80b] Xerox Corporation "Welcome to Alto Land: Stanford ALTO User's Manual", Xerox Corporation, 1980 User manual for Xerox Alto personal computer. Includes descriptions of "chat window" user interface in FTP GUI. [Yacyk78] Yacyk, J. "Alphabetic Handprint Reading", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-8 No 4, pp , April 1978 Constraints do not work well on handprinting in practice: Users tend to omit serifs from O and S in ANSI74 standard of constrained writing styles. Review of OCR for ANSI-standard handprints, shows Universal Subset standard neographic shapes for hand-printed characters and symbols. Tersoff paper is on same page [YamamotoK78] Yamamoto, K. and Mori, S. "Recognition of hand-printed characters by outermost point method", Proceedings of Fourth International Joint Conference on Pattern Recognition, Kyoto, Japan, November 7-10, 1978, pp Cited in Kahan87 [YamamotoK80] Yamamoto, K. and Mori, S. "Recognition of Hand-printed Characters by an Outermost Point Method", Pattern Recognition, Vol 12., pp , March 1980 Similar to BLRTs chain codes, but uses 16 directions, not 4 Refers to dis-ambiguation of Katakana and Roman characters [YasuharaM76] Yasuhara, M. "On Distinctive Features in Character Recognition -- Some Evidence from Reactiontime Measurements", Working Paper 102, Cognitive Information Processing Group, Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, August 15, :26:16 PM]

74 ( ) History of Pen and Gesture Computing Mentions lack of consensus on what features to use in any given pattern recognition problem [YasuharaM77a] Yasuhara, M. and Yasumoto, Yasuhiko "Measurement of Handwriting Pressure - Pressure- Sensitive Pen", Annual Report of RLCS Avoid feature extraction in signature verification by time-template matching: compare elastic matching [YasuharaM77b] Yasuhara, M. and Oka, M. "Signature Verification Experiment Based on Nonlinear Time Alignment: A Feasibility Study", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-7 No 3, pp , March 1977 Signature verification using pressure-sensitive stylus on tablet: not clear that it uses pressure information, focuses on elastic time matching to allow for speed/velocity variations while writing. [YasuharaM78] Yasuhara, M. and Kuklinski, T. "Category boundary effect for grapheme perception", Perception and Psychophysics, Vol 23 No 2, 1978, pp Discrimination function peaks at the category boundary, just as speech does Boundary classification for character recognition Ted Kuklinski and Yasuhara: says say "physical attribute" and "functional attribute", since "feature" is used sloppily Shows ambiguity of feature "leg" along character trajectory, instead of character along feature trajectory Cultural bias: l vs r in speech by Japanese, and by Americans (extension to characters: European 7 vs F (?)) [Yoshida78] Yoshida, Masumi; Iwata, Kiyoshi; Yamamoto, Eiichiro; Masui, Takeshi; and Kabuyama, Yukikazu "Pattern Recognition Processing System", United States Patent 4,105,998, assigned to Fujitsu Limited, Kawasaki, Japan, August Recognizing handwritten characters by their intersections within a rectangle with horizontal and vertical reference lines; used the distances and positions of the intersection points, such as to get at the internal "open" space within a character [Yoshida80] Yoshida, M. et al "Recognition System for Design Chart Drawn on Section Paper", Proceedings of 5th International Joint Conference on Pattern Recognition, 1980, pp Cited in Kato80: user-interface for prettying up drawings [YouKC79] You, K.C. and Fu, K.S. "A syntactic approach to shape recognition using attributed grammars", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-9, 1979 biblio75 (Prev page) Main Page (Next page) biblio83 According to legend, Walt Disney never visited Disneyland. Der Weise sagt uns, dass wir sie an ihren Taten werden erkennen, und nicht an ihren Worten.... Aber was ist, wenn das was sie tun, ist viel reden? 8:26:16 PM]

75 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1981 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:25 EDT Mirrors of this page can be found at: (Generally most up-to-date) biblio80 (Prev page) Main Page (Next page) biblio85 [ABAJournal81a] ABA Journal "Computer Add-Ons: Micropad handwriting terminal", ABA Journal, May 1981, Vol 67, page 625 Product information announcement about Micropad handwriting-recognition terminal (see file) from Micropad, Inc.: Micropod is a local or remote input terminal which captures handwritten data at the time of writing, translating It into machine readable form. [ANSI81] ANSI "ANSI X3H31 Status Report on the H31 Strawman Proposal on Input Functionality", Document X3H31/31-11R1, April 20, 1981 [ActaPsychologica83] Acta Psychologica "Special issue on motor aspects of handwriting", Vol 54, 1983 [Agui81a] Agui, T., Matsubara, K., and Nakajima, M. "Sequential Computer Processing of a Collection of Closed Curves and Its Application to Pattern Recognition", Trans. of the IECE of Japan, Vol E64, No 10, October 1981, pp (abstract only) Chain-code transformation, for matching similar curves [Amin82] Amin, Adnan, and Masini, Gerald "Machine recognition of cursive Arabic words", SPIE Vol 359, Applications of Digital Image Processing IV, 1982, pp Dictionary lookup for word-by-word handwriting recognition of Arabic Segmentation: some Arabic letters are discontinuous, so Arabic words are 1 to 7 strokes (not counting dots?) Handwriting spelling dictionary look up by number of strokes, number of dots, number of intersections in Arabic Cited in FoleyJD82 Input model: precursor to PHIGS and CGI, postcursor to SIGGRAPH CORE? [Anson82] Anson, E. "The Device Model of Interaction", Computer Graphics, Vol 16 No 3, July 1982, pp Critique of CORE graphics standard, need for composite devices. Compare with proposal by Pencept for handwriting recognition input to PHIGS standard. Model supports multiple (simulated) devices, two-handed input, contrasted with ping-pong input of one device at a time. Refers to user-interface of simulating a virtual soft keyboard /function buttons on a tablet [Apollo83a] Apollo Computer "Getting Started With Your Domain System (Apollo)", Apollo Computer, 1983 General user manual for Apollo workstation. Heirarchical file system, windows. No citation to Unix. Touchpad included on workstation keyboard. 8:26:20 PM]

76 ( ) History of Pen and Gesture Computing [Apple83a] Apple Computer "Apple Modem 300/1200 User's Manual", Apple, Inc (hardcopy book) Modem command set for Hayes-compatible Apple modem [Applicon83] Applicon "IAGL User's Guide", Applicon Incorporated, Burlington, Massachusetts, June 1983 cited in RubinSM84: Ledeen recognizer? [ArakawaK83a] Arakawa, K. "On-line recognition of hand-written characters -- Alphanumeric, hiragana, katakana, kanji", Recognition, Vol 16 No 1, pp 9-16, Published earlier as (Arakawa78) Fourier coefficients of strokes as feature points For Roman alphabet, 29 single-stroke and 7 two-stroke character shapes For Hiragana alphabet, 10 single-stroke and 18 two-stroke character shapes, stroke, 5 4-stroke [BabbGR82] Babb, Gerald R. and Kuklinski, Theodore T. "Pattern Algorithm Permits Freehand Printed-Data Entry", Computer Technology Review, Winter 1982 Shows basic features of first Pencept product: single-stroke and multi-stroke forms, boxed input [Badie82] Badie, K. and Shimura, M. "Machine Recognition of Roman Cursive Scripts", Proceedings of the 6th International Conference on Pattern Recognition, pp 28-30, 1982 Refers to similarity/variability of loop and arc in script handwriting Script recognition using clock-wise vs counter-clockwise loops and arcs Refers to corner(cusp)/loop transition in script writing [Bahl81] Bahl, L.R. and Cocke, J. "Font-Independent Character Recognition by Cryptanalysis", IBM Technical Disclosure Bulletin, Vol 24 No 3, August 1981, pp Cites Baum-Petree algorithm for decoding Font- and language-independent recognition by doing cryptanalysis on whatever categories and arbitrary recognition comes up with? [Bahl83] Bahl, L.R., Jelinek, F., and Mercer, R.L. "A Maximum Likelihood Approach to Continuous Speech Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-5 No 2, March 1983, pp Realistic testing: artificial tasks vs natural tasks (for a priori grammar for speech) Speech: acoustic/phonetic encoding vs communication theory model [BarachDR81] Barach, David R, Taenzer, David H, Wells, Robert E. "Design of the PEN Video Editor Display Module", ACM conference proceedings: ACM /81/0600/0130 Video text editor to emulate different computer keyboard terminals, with separation of data and display modules: does not involve pointing/pen or character recognition input (Copy provided by Gary Odom) [BarkerPG82] Barker, P.G. "Data base interaction using a hand print terminal", International Journal of Man- Machine Studies, Vol 17, 1982, pp DCR Dynamic Character Recognition devices allow several novel kinds of user interactions / interfaces. Says needs much higher resolution digitizer touch-pad than is available. [BartelsRH83a] Bartels, Richard H.; Beatty, John C.; Booth, Kellogg S; and Field, Daniel E. "Computer Graphics Laboratory Fall 1983 Review, Univ. of Waterloo", University of Waterloo, Computer Science Department, Report CS-83-33, December 1983 GTCO Digi Pad 5 Pressure Pen Tablet; comments on lack of smarts used in microprocessors for functionality of pointing devices: example is buffering of data in pointing device to avoid data over-run from host buffering-- does not comment on latency. Labanotation input for dance transcription (Benesh editor) [BewleyWL83a] Bewley, William L.; Roberts, Teresa L.; Schroit, David; and Verplank, William L. "Human Factors Testing in the Design of Xerox's 8010 "start" Office Workstation", Proc. CHI '83, December 1983, pp User interface usability testing of Xerox Star GUI, primarily on use of mouse buttons, 1/2/3-button mouse: separate button for drag/copy/select, multiple clicks (double click, etc.) Drawthrough gesture for selection: not quite press-and-hold. [Biswas81] Biswas, Prasenjit and Majumdar, Arun K. "A Multistage Fuzzy Classifier for Recognition of Handprinted Characters", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-11 No 12, December, 1981, pp Test results on 12 of the 35 Devanagari alphabet handwritten characters (all work done by hand: did not have a real computer?) Fuzzy set classifier for Devanagari (Indian) handwriting recognition Data collection: tested on the 70% "satisfactory" sample writing sheets: bias in sample? points out subjectiveness of training? (pairwise comparison?) syntactic method only applied to handwritten characters which were confuseable [Blesser83] Blesser, B. "Multistage Digital Filtering Utilizing Several Criteria", United States Patent 4,375,081, February 22, 1983 Low-pass digital filter for a tablet to pre-process handwritten character to eliminate wobble/digitization noise before recognition [Bozinovic82] Bozinovic, Radmilo and Srihari, Sargur N. "A string correction algorithm for cursive script recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol 4, November 1982, pp Handwriting cursive script recognition using channel model of dictionary: allows for splitting, merging, and substitution (segment parsing errors). Cited in Bozinovic89 for context: probabilistic spelling corrector based on a distance metric for strings. 8:26:20 PM]

77 ( ) History of Pen and Gesture Computing [BritishMicro82a] British Micro "Operating Guide to GRAFPAD", British Micro, 1982 User guide for GRAFPAD, low-cost graphics tablet for ZX Spectrum early PCs. Resolution 256x320 (16x20 of boxes of 16x16 points), stylus with tip switch. Printed overlays for virtual keyboard (not on-screen), application menus. See also CPCWiki reference. [BrodyH83a] Brody, Herb "Machines that Read Move Up a Grad", Hight Technology, Feb 1983, pp 35-ff Review of commercial OCR products and applications (document sorting), including Kurzweil, [BrownF83] Brown, Frances "The acquisition of handwriting in the UK", Oral presentation at the Criminalistic Institute in Prague, Written form available at Report on handwriting styles taught and learned in the U.K. Points out that there is no single style of handwriting taught in copybooks in the U.K., unlike some other countries. Advisor, Tom Davis, comments on the diversity of writing styles in signature verification and the detection of forgeries [BrownMK83] Brown, M.K. and Ganapathy, S. "Preprocessing Techniques for Cursive Script Word Recognition", Pattern Recognition, Vol 16 No 5, pp , 1983 adaptive/trainable recognition, suggests "closed loop" verification (feedback to user) is helpful Much DCR/CSR handwriting/gesture recognition research is commercial/proprietary, therefore not published. DCR (gesture/handwriting input) picking up as result of human factors in Man/Machine interface. Writer (author) independent recognition would be more desirable than present trained recognition. In development, necessary to hand-correct for real-world errors in input data collection. [BrownRM83] Brown, Robert M., and Cheng, C.F. "Optical Character Recognition for Automated Cartography: The Advanced Development Handprinted Symbol Recognition System", Naval Ocean Research and Development Activity, NSTL Station, Mississippi, Report No NORDA-TN-187, March 1983 NTIS citation index [Buckle81] Buckle, Derek and Strand, Timothy D. "Processing of Information", United States Patent 4,262,281, assigned to Quest Automation Limited, Dorset, England, April 14, 1981 Quest Automation / Datapad product patent on handwriting recognition: Claims on alignment/registration of paper on digitizer tablet by marking on pre-printed alignment targets. Pressure-sensitive tablet with floating hand rest to keep hand from pressing on tablet. See also Micropad [BurrDJ81a] Burr, D.J. "Elastic matching of line drawings", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-3, March, 1981, pp Handwritten sketch/character input recognition: refers to IEEE handwriting data base collection 1.2.4: (24x36 binary OCR) Like Greenberg77, throws out handwriting samples from IEEE Database as "poorly written" [BurrDJ83a] Burr, D.J. "Designing a Handwriting Reader", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-5 No 5, pp , September 1983 Context by matching grouped letters into words using Unix "spell". Separate "training dictionary" for each user: single-stroke characters, discrete characters. Segmentation by pen-lift only, therefore unistroke only: "t" "i" etc. must be made with one stroke. [BuryKF81a] Bury, Kevin F.; Boyle, James M; Evey, R. James; and Neal, Alan S. "Windowing vs Scrolling on a Visual Display Terminal", Proc. CHI '82, pp Study of windowing vs scrolling on displays: difference is direction: whether upward motion moves apparent window up, or apparent document up. Most subjects preferred windowing, but scroll users strongly preferred. Input was cursor arrow keys, not mouse or tablet: so not clear on effects of having a scroll bar or a flick/scroll gesture. [Buxton82] Buxton, William "An Informal Study of Selection Positioning Tasks", Graphics Interface '82, pp 323-ff trainable, short-hand single-stroke symbols for graphical shapes, graphics editing: Unistroke, terminated by pen lift at the end. Gestures are checkmark, downward stroke, and upwards stroke. [Buxton83] Buxton, William, Fiune, Eugene, Hill, Ralph, Lee, Alison, and Woo, Carson "Continuous hand-gesture driven input", Proceedings of Graphics Interface 83, page , 1983 Gesture input: tap/circle gesture. Says velocity-sensitive gestures difficult Cited in Rhyne86: Digitizer stylus must be held vertical to get good results -- reference to stylus tilt problem, often miss-called parallax. [Buxton83a] Buxton, William "Lexical and Pragmatic Considerations of Input Structures", Computer Graphics, January 1983, pp 31-ff Lexical (chunking and phrasing) aspects of compound gestures [Buxton83b] Buxton, William "Towards a Comprehensive User Interface Management System", Computer Graphics, Vol 17, No 3, July 1983 MENULAY/MAKEMENU system: User-interface editor, user can sketch user interface and then associate actions with user interface elements. Uses four-button puck on graphics tablet. [BuxtonWB83c] Buxton, William "Etch: A Study in Marking-Based Interaction", University of Toronto, 1983: available at billbuxton.com 8:26:20 PM]

78 ( ) History of Pen and Gesture Computing Video demonstration to accompany "Continuous Hand-Gesture Driven Output". Complexity of marking menus can be reduced by structuring the dialogs. [BuxtonWB83d] Buxton, William "Menulay: A UIMS for Rapidly Sketching Prototyping and Implementing User Interfaces", University of Toronto, 1983: available at billbuxton.com Video demonstration to accompany "Towards a Comprehensive User Interface Management Systems. System for defining user interfaces using sketching and marking menus. [Byte82a] Williams, Greg "Title Article: The IBM Personal Computer -- A Closer Look at the IBM Personal Computer", BYTE Magazine, Vol 7 No 1, January 1982 Overview of IBM personal computer. IBM support both CP/M-86 and MSDOS. "A Closer Look at the IBM Personal Computer" (Gregg Williams). IBM Basic (GW/Basic). Also articles on Cromix (Unix OS clone), Atari graphical smooth scrolling implementation. HIPAD tablet digitizer from Houston Instruments, with keyboard simulation. [Byte83a] BYTE Magazine "Special Issue: Inside the IBM PC", BYTE Magazine, Vol 8 No 11, November 1983 Detailed hardware and software description of features of the IBM personal computer. MS/DOS 2.0 installable drivers, background tasking. Pencept PenPad text/graphics with handwriting recognition. IBM Kana/Japanese keyboard / IBM 5550 system keyboard. "Big Blue goes Japanese" Has pictures of the complexity of keyboards for Japanese and Chinese characters. Article on Concurrent CP/M operating system [CTS81] CTS Recognition "Telepad product information", Golden Square, London W1R 3AG, England, 1981 Early Micropad-like British handwriting recognition product. [CalComp83] CalComp "CalComp 2000 Series Digitizer Operator's Manual", , page 15, CalComp Incorporated, January 1983 [Carau81] Carau, F., Hetzel, H. and Tremblay, M. "Travelling Wave Digitizer", United States Patent 4,255,617, assigned to Hewlett-Packard Co, Palo Alto, California Digitizer measuring X and Y sequentially in time. Mentions velocity correction for bowing of diagonal lines, corrections for non-orthogonality, course and fine measurements in two stages. X/Y Grid, puck with magnifying glass shown. [CardSK83a] Card, Stuart K.; Moran, Thomas P.; and Newell, Allen "The Psychology of Human-Computer Action", Lawrence Erlbaum Associates, 1983 (hardcopy book) User interfaces discussed in terms of keyboard interactions. One section on Grafcon tablet: section on pointing/selection speed only compares joystick vs. mouse. [Carey83] Carey, Tom "User Differences in Interface Design", IEEE Computer, November 1982, pp 14-ff Early essay on user interface design for varied users: range to be dealt with is not just beginner/intermediate/experienced users, also range of learning styles (hands-on versus read the documentation, visual learners versus text learners). [Carroll82] Carroll, John M. "The Adventure of Getting to Know a Computer", IEEE Computer, November 1982 User interfaces better if friendly: no unfriendly error messages. Also mentions discoverability in user interface design. [Casey82] Casey, R.G. and Nagy, G. "Recursive Segmentation and Classification of Composite Character Patterns", Proceedings of the 6th International Conference on Pattern Recognition, Munich, Germany, October 1982, pp Combine character segmentation with classification in adaptive decision tree Optical resolution good enough for recognition may be too low to segment characters (like cursive/connected vs discrete characters segmentation): some touching characters in OCR can only be segmented by recognizing component characters [Casio83] Casio "Module No. 658 Wrist-watch Controller", Casio, 1983 Touchscreen controller for Casio wristwatch. Electronic file includes additional material on VDB-1000, Casio Databank with small keyboard on wristwatch. Very early Smart-Watch. User manual for Casio wristwatch: may be the model with touchscreen input and handwriting recognition, manual describes small keyboard input. [ChalkBoard83a] Devlin, Joe "The legend of the pad of power", Creative Computing, Vol 9 No 10, October 1983, page 52 Chalkboard Power Pad: touch-sensitive membrane graphics tablet responding to more than one touch at a time, can image contact of entire hand. Resolution 120x120. Sold as educational toy. multi-touch capable? [ChalkBoard83b] Crowell, Gregory William "Chalkboard PowerPad graphics tablet", Chalkboard Inc, 1983: at Personal Computer Museum, Brantford Ontario, Canada (2011) Chalkboard Power Pad: pressure-sensitive membrane graphics tablet responding to more than one touch at a time. Resolution 120x120. Sold as educational toy. Pressure-switch touch graphics tablet, 14,400 sensor points for 10x10 resolution on 12x12 inch surface. "Has no problem resolving simultaneous multiple inputs" (multi-touch), "software stretching" to interpolate (?) to higher resolution. 8:26:20 PM]

79 ( ) History of Pen and Gesture Computing [Cognex83a] Cognex Corporation "Cognex Corporation presents DataMan", Cognex Corporation, 1983 Date approximate: OCR product for reading serial numbers and other identifying numbers in manufacturing and shipping, such as product IDs on boxes. [ComputerGW82] Computer Graphics World "Digitizer Survey", Computer Graphics World, July 1982, pp Vendor survey in 1982 for digitizing tablets, video scanner digitizers. [Computerworld81a] Computerworld "Micropad: Data Entry Unit Accepts Writing", Computerworld, July 6, 1981, p. 52 Announcement of Buffered Micropad handwriting recognition input terminal, 512 characters, built-in oneline LED display. [Computerworld82a] Computerworld "Image Data Tablet System", Computerworld, January 18, 1982, page 14 Image Data Tablet System: Tablet and monitor with handprint (handwriting) recognition, mathematical calculation, graphics mode input, on-tablet keyboard icons. May be Pencept PenPad 200? [ConklinD83] Conklin, Dick "PC Graphics (excerpt)", Wiley IBM PC services, ISBM Early reference to digitizing tablets with IBM PC: rotational transformation, digitizer resolution much higher than display: typical resolution of tablet 2048 points (sic) [CooperLN81a] Cooper, Leon N., and Elbaum, Charles "Information Processing System using Threshold Passive Modification", United States Patent 4,254,474, March 3, 1981, assigned to Nestor Associates, Stonington, Connecticut Nestor Graphics patent [CooperLN82a] Cooper, Leon N. and Elbaum, Charles "Curve Follower", United States Patent 4,319,331, March 9, 1982, assigned to Nestor Graphics, Providence, Rhode Island Nestor Graphics patent Curve follower to turn OCR data into DCR character data [CooperLN82b] Cooper, Leon N., Elbaum, Charles, and Reilly, Douglas L. "Self Organizing General Pattern Class Separator and Identifier", United States Patent 4,326,259, April 20, 1982, assigned to Nestor Graphics, Providence, Rhode Island., also European Patent Application Nestor Graphics patent [Cordella83] Cordella, L.P. and Sanniti di Baja, G. "Structural description of silhouettes", Proceedings of the 3rd Scandinavian Conference on Image Analysis, Copenhagen, Denmark, 1983, pp Cited in Arcelli85: about line-thinning and reconstruction [Cox82] Cox, C.H. III, Coueignoux, P., Blesser, B., and Eden, M. "Skeletons: A Link Between Theoretical and Physical Letter Descriptions", Pattern Recognition, Vol 15 No 1, PP 11-22, 1982 Barry's group: functional attribute (cognitive) vs synthetic (generative) Deal with embellishments separates from base pattern (in OCR) [CraneHD82a] Crane, H.D. and Wolf, D.E. "Dynamic Creation of Signatures", United States Patent 4,344,135, August 10, 1982, assigned to Stanford Research Institute, Menlo Park, California Hew Crane: amendment to Crane79: patent 4,156,911 (?) Human-reading for signature verification [CraneHD83a] Crane, H.D. and Ostrem, J.S. "Automatic Signature Verification Using a Three-Axis Force-Sensitive Pen", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-13 No 3, May 1983, pp Writing pen with obstructed view: accelerometer (no touch screen) [DahlJP81a] Dahl, Jerome P.; Epley, Phillip R.; and Fox, Jon E. "Program Controlled Capacitive Keyboard Variable Threshold Sensing System", United States Patent 4,305,135, December 8, 1981 Capacitive key matrix keyboard, references touch-sensing circuit in IBM journal [DamerauFJ82a] Damerau, Frederick J. "Guess-ahead feature for a keyboard-display terminal data input system", United States Patent 4,330,845, May 13, 1982 Auto-completion of typed text: if what is typed so far matches text in a list, monitor shows rest of that text without advancing cursor, user can type over or hit a function key (right arrow) to accept. Compare with auto-completion in MAC LISP, word prediction in accessibility on-screen keyboard software? [DenningDE81a] Denning, Dorothy E. and Sacco, Giovanni Maria "Timestamps in Key Distribution Protocols", CACM vol 24 no 8, August 1981, pp Use of timestamps in key-exchange algorithms and signatures to prevent re-plays. (Compare with use of nonce). [DenningDE82a] Denning, Dorothy Elizabeth Robley "Cryptography and Data Security", Addison-Wesley, 1982 Textbook on cryptography and security -- security primarily access controls (Bell-La Padula, etc.) Certificate and public/private key PKI distribution. Hamming encoding. [DooijesEH83a] Dooijes, E.H. "Analysis of Handwriting Movements", Acta Psychologica, Vol 54, 1983, pp Frequency-domain analysis of handwriting movements in X and Y: Lissajous analysis [DosterW83a] Doster, W. and Oed, R. "Zur Bildanalyse bei der Handschriftlichen Direkteingabe", Proceedings of Mustererkennung 1983, October 11-13, 1983, Karlsruhe, West Germany Best description of AEG's segmentation/parsing algorithm for handwritten characters Blithely claims that rasterization of dynamic on-line character data would reduce this with OCR to the same problem (but the 8:26:20 PM]

80 ( ) History of Pen and Gesture Computing OCR segmentation problem is harder!) [DosterW83b] Doster, W. and Schuermann, J. "A Step Towards Intelligent Document Input to Computers", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 19-23, 1983, Washington, D.C., pp Scan in documents with OCR, use on-line handwriting recognition user-interface to edit them [Downton83] Downton, A.C., Baker, R.G., Lewis, S.M., and Cooper, P.J. "Readability measurements of Palantype transcription for the Deaf", International Journal of Man-Machine Studies, Vol. 19, pp , 1983 [DunnK81a] Dunn, K. "Choose Digitizer Technology and Features to Suit Applications", Computer Technology Review, Fall/Winter 1981, pp Many limiting characteristics of digitizers (width of pencil line, etc.). List of what can be spec'ed for a digitizer performance (but not the trade-offs). [DunnK81b] Dunn, K. "Understanding digitizer resolution and accuracy", Mini-Micro Systems, December 1981 Differential linearity error is a digitizer's worst enemy; Clumping and stretching of digitizer co-ordinates for X-vs-X non-linearity; Digitizer accuracy vs digitizer stability; General review of digitizer characteristics and proper specifications [Elliot82] Elliott, B.J. "Apparatus for Determining Pen Acceleration", United States Patent 4,345,239, August 17, 1982, assigned to International Business Machines, Armonk, New York Capacitive sensor for measuring pen/stylus acceleration for signature verification [Embley81] Embley, D.W. and Nagy, G. "Behavioral Aspects of Text Editors", A.C.M. Computing Surveys, Vol 13 No 1, March 1981, pp Cites gaps in human factors/engineering and cognitive psychology literature re text editing user-interface; human factors of using digitizing tablets for text editing Cites "taste for federal funding" as reason some human factors studies have skipped various interests: research selection bias Users frequently will not use mouse in text editing: (we say "3-handed monkey" effect): three-handed monkey Tablet: pointing with light pen fatigueing, and not accurate enough [EvansKB81] Evans, Kenneth B.; Tanner, Peter P.; and Wein, Marceli "Tablet-based Valuators that Provide One, Two, or Three Degrees of Freedom", Computer Graphics, Vol. 15 No. 3, August 1981, pp Digitizer tablet simulation of virtual input devices: turntable, multiple number wheels, three-axis trackball [EventoffFN81a] Eventoff, Franklin N. and Christiansen, M. Tyrone "Multi-function Touch Switch Apparatus", United States Patent 4,268,815, May 19, 1981 Dual-function pushbutton switch: one pair of layers is simple on/off to turn on power of (MOS) device, second pair of layers is force/pressure-sensitive transducer (FSR material). [EventoffFN82a] Eventoff, Franklin N. "Electronic Pressure Sensitive Transducer Apparatus", United States Patent 4,314,227, February 3, 1982 Force/pressure-sensor using thin film of material with microscopic particles of conducting material (forcesensitive resistor). Does not depend on resilience/elastomeric/rubbery of material for a restoring force: uses contact resistance. [Fairhurst82a] Fairhurst, M.C. "Image Characteristics as Assessment Criteria for an Electronic Writing Aid", Proceedings of International Conference on Man-Machine Systems, July 1982, pp Claims better test results for machine recognition than for human reader (!) [Fairhurst82b] Fairhurst, M.C, and Maia, M.M. "An Approach to Machine Reading of Text with a Memory-based Character Recognition System", Colloquium Proceedings "Coding of Documentary Information", University of Kent, Canterbury, England, May 1-2, 1982 Discusses pairwise discriminators to reduce memory requirements for OCR of typed text: Similar to Pencept? [FidlerR81a] Fidler, Roger "Newspapers in the Year 2000: Videotex services will become mature businesses", The Society for News Design / APME, Reprinted January 14, 2010 Speculative article on use of tablets / flat-screen displays as electronic newspapers, continuous updating. Refers to tactile controls (touchscreen). Compare with Boston Globe / NYTimes electronic distribution on tablets [FieldsC83] Fields, Craig I. "Virtual Space Teleconference System", United States Patent 4,400,724, August 23, 1983 Not a whiteboard system: video conferencing arrangement with each user facing one monitor and one video camera per other user. [Filipski81] Filipski, Alan J. "Critical-point Representation of Hand-printed Numerals", Proceedings of International Conference on Cybernetics and Society, IEEE 0360/8913/81/ , October, 1981, Atlanta, Georgia, pp % correct on handprinted OCR recognition: skeletonizing, "critical point" strokes with initial and final slope features for 18 types of stroke segments. Admits to and spells out several weaknesses: 4 vs 9, patching OCR skeletonization breaks Handwritten OCR recognition: simple template matching works well if 8:26:20 PM]

81 ( ) History of Pen and Gesture Computing feature extraction is good Critical points for features and segmentation in OCR handwriting: points of high curvature, then Freeman chain codes Refers to Knoll Database of hand-printed numerals, 21x25 binary grid, IEEE Pattern recognition data base 1.2.2) [FoleyJD81] Foley, J.D., Wallace, V. and Chan, P. "The Human Factors of Interaction Techniques", George Washington University, Institute for Information Science and Technology Technical Report GWU-IIST-81-03, Washington, D.C., 1981 Precursor to FoleyJD84 [FoleyJD82] Foley, J.D. and VanDam, A. "Fundamentals of Interactive Computer Graphics", Addison-Wesley, Reading, Massachusetts, 1982 (hardcopy book) Three-point calibration, general graphics reference Comparison of tablet and mouse "locator" devices, absolute position versus relative motion. Section 5.3 "Technique for simulating logical devices": "locator device simulation" / virtual devices, simulated keyboard, character recognizers. Says Ledeen recognizer simplified version of Teitelman 1964, described in Newman 199 [Freyd83] Freyd, J.J. "Representing the dynamics of a static form", Memory and Cognition, Vol 11, 1983, pp Tappert's bibliography [Fu81] Fu, K.S. "A survey on image segmentation", Pattern Recognition, Vol 13, 1981, pp 3-16 Fu86/Fu83 [Fukushima83] Fukushima, K., Miyake, S., and Ito, T. "Numeral Character Recognition by the Algorithm of the Neocognitron", Trans. of the Institute of Electronic and Communications Engineers of Japan, Vol J66D No 2, February 1983, pp NTIS index: neural network for handwritten numerals recognition, using a scanner: learning-with-a-teacher better than learning-without-a-teacher (training set makes a difference, but what?) [Furuta82] Furuta, Richard, Scofield, Jeffery and Shaw, Alan "Document Formatting Systems: Survey, Concepts, and Issues", Computing Surveys, Vol 14, No 3, September 1982, pp 417-ff Very little treatment of UI issues or pointing devices: primarily deals with keyboard commands and display of monofont text [GTCO82] GTCO "Digi-Pad 5 Family Data Sheet", DP5-L , GTCO Corporation, Rockville, Maryland, 1982 GTCO Electronic grid digitizer, serial interface: pressure-sensitive pen/stylus, 16-button puck/cursor [Gander83] Gander, Stephen Joseph "A proposed method of handwriting recognition", S.B.E.E. Thesis, M.I.T., 1983 (abstract only) Not from Blesser group [Gehani82] Gehahi, N. "The Potential of Forms in Office Automation", IEEE Trans. Communication, Vol 30 No 1, January 1982 Gives many reasons for using forms as fundamental office automation model. (taken from Hekmatpour86). Use of forms eases transition from manual to office automation. [Geyer81] Geyer, L.H., and Gupta, S.M. "Recognition/confusion of dot matrix vs conventional font capital letters", Perception and Psychophysics, Vol 29, 1981, pp Cited in Suen86 Suen86 cites this on what matrix resolution needed for OCR on hand-print [GlickmanD82a] Glickman, David, Greanias, Evon C., Repass, James T., and Rosenbaum Walter S. "Stem Processing for Data Reduction in a Dictionary Storage File", United States Patent 4,342,085, assigned to International Business Machines Corporation, Armonk, New York, July 27, 1982 Stem processing for data reduction in dictionary storage file storing word list file with prefix and suffix truncated so that only unique root element remains [Goodale83] Goodale, T.S., Goyal, S., and Litvin, Y. "Designing a Text Editor with Graphic and Handwritten Input", Report TR , GTE Laboratories Incorporated, 40 Sylvan Road, Waltham, Massachusetts 02254, November 1983 (partial copy) Cited in Litvin89 Gesture recognition for a text editor? Handwritten text input? Gesture-based user interface with handwriting recognition for editing text with electronic ink: first reference for prior art? Early reference to visual parallax on electronic ink with integrated tablet/display. [Gould83] Gould, J., Conti, J., Tovanyecz, T. "Composing Letters with a Simulated Listening Typewriter", CACM, pp , Vol 26 No 4, April 1983 Simulation of "perfect" speech recognition, showing that there is indeed a user-interface problem in addition to recognition problems. [Greanias82] Greanias, Evon C. and Yhap, Ernesto F. "Chinese/Kanji On-line Recognition System", United States Patent 4,365,235, December 21, 1982, assigned to International Business Machines Corporation, Armonk, New York Chinese recognition by recognizing component strokes/radicals Refers to "spelling" of Chinese/Kanji: order strokes/radicals are written in States that strokes are classified into 42 categories for segmentation, and that Chinese/Kanji characters are made up of only 72 basic symbol elements/alphabet Cites problem with 8:26:20 PM]

82 ( ) History of Pen and Gesture Computing sonic/acoustic digitizers: sensing point offset from writing point If Kanji/Chinese character is not recognized, user can add it to the prototype set for recognition on the fly [Greer83] Greer, K.L., and Green, D.W. "Context and Motor Control in Handwriting", Acta Psychologica, Vol 54, 1983, pp Digitizer specifies " on resolution, but also accuracy [Gu83] Gu, Y.X., Wang., Q.R., and Suen, C.Y. "Application of a Multilayer Decision Tree in Computer Recognition of Chinese Characters", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-5 No 1, January 1983, pp Features for Chinese character recognition are Walsh coefficients, projected onto X and Y axes: character alignment (normalization), 99.5% accuracy on 3000 characters OCR on Chinese/Kanji characters Using a binary decision tree for pattern recognition of a large number of classes (Chinese characters OCR) Performance: noisy characters only: clean characters do not occur in real life [GurolIM83a] Gurol, I. Macit and Bang, Gary M. "Touch Sensitive Transparent Switch Array", United States Patent 4,423,299, December 27, 1983 Transparent switch matrix for touchscreen, conformable to a display: grid of conductors separated by spacer dots, pressing on top flexible sheet/membrane closes switch of two conductors. [HP83a] Hewlett-Packard "HP-150 touchscreen Personal Office Computer product", (various) File contains reference for HP-150 touchscreen computer: MS/DOS personal computer (8088) with infrared touchscreen on 9-inch video display. Also known as Touchscreen MAX. No OS support, applications had to be written specifically for touchscreen input. [Haber81] Haber, R.N. and Haber, L.R. "Visual components of the reading process", Visible Language, Vol XV No 2, 1981, pp Cited in Bozinvic89 Bozinovic cites for human recognition by outline of word (as shown in WrightG52) [Hagita83] Hagita, N., Naito, S. and Masuda, I. "Handprinted Chinese Characters Recognition by Peripheral Direction Contributivity Feature", Trans. of IECE of Japan, Vol J66D No 10, October, 1973, pp (in Japanese: abstract only) OCR for Chinese using features of stroke structure, complexity, direction, connective relation, and relative location for 95.4% of block-style handprinted data [Hanaki81a] Hanaki, S. and Yamazaki, T. "On-line recognition of handprinted Kanji characters", Pattern Recognition, Vol 12, 1980, pp Tappert's bibliography -- Chinese recognition [Hanaki81b] Hanaki, S., Temma, T., Yoshida, H., Arakawa, T., Suziki, M., Seki, T., and Kikuchi, Y. "Online Realtime Character Recognition System", Trans. of IECE of Japan, Vol E64 No 5, p 374, May 1981 (Abstract only) Science Citation Index Handprinted recognition: feature is break into piecewise segments, compare with decision tree: 32 handwriting terminals on one computer (cluster) [Hemenway82] Hemenway, K. "Psychological Issues in the Use of Icons in Command Menus", Proc. Human Factors in Computer Systems Conference, Washington, D.C., A.C.M., March 1982, pp Cited in FoleyJD84 [HewlettPackard83a] Hewlett Packard "Hewlett Packard HP-150 Touchscreen Computer", wikipedia.org, fetched 2011 Early (DOS-age) touchscreen computer product. [HillisWD81a] Hillis, William Daniel "Active Touch Sensing", M.I.T. A.I. Memo 629, April 1981 High-resolution touch sensor, several hundred sensing points per square inch. Uses grid/matrix of anisotropic elastomeric material: crosspoint problem addressed by grounding all other lines than the ones being sensed, so that no current flows. Shear forces determined by force needed to move the probe across whatever it is touching. [HillisWD82a] Hillis, W. Daniel "A High-Resolution Imaging Touch Sensor", Int'l Jnl. of Robotics Research, June 1982, Vol 1 No 2, pp (Abstract only) Imaging touchpad sensor for robotics using small pressure/force sensors. [HollerbachJM81a] Hollerbach, John M. "An Oscillation Theory of Handwriting", Biological Cybernetics, Springer- Verlag, Vol 39, 1981, pp (see also Hollerbach78) Unclear fancier curve fitting to handwriting models gives any more insight Maximum writing speed/velocity is 25 mm/sec Acceleration peaks in handwriting not just a tablet/pen artifact Variation in slant in handwriting (vertical) is about 10 degrees Cusp/loop: substitution of clockwise vs counter-clockwise motion in handwriting Variability of corner shapes for single writer Says reduction in concentration on handwriting needed to think and write at same time [Honeywell81a] Honeywell Information Systems "Level 68 Introduction to Programming on Multics", Honeywell Information Systems AG90-03, July 1981 Multics operating system programming tutorial: special section on dynamic linking. All files are memorymapped into address space, no separate I/O. Mentions direct file types stream/sequential: stream is ascii 8:26:20 PM]

83 ( ) History of Pen and Gesture Computing file? APIs for access control ACLs. [Honeywell83a] Honeywell Information Systems "MULTICS Common Commands", Honeywell Information Systems G58-0 February 1983 Multics operating system CLI command-line commands: set_acl; copy command preserves ACLs access control lists; send_mail over ArpaNet. [Honeywell83b] Honeywell Information Systems "Inter-MULTICS File Transfer Facility Reference Manual CY73-01", Honeywell Information Systems Order Number CY73-01, December 1983 Multics file transfer facility: describes I/O Daemon process. On target system for file transfer, ring access must be at higher numbered access level (more restrictive), I/O daemon must have ACL access rights, user on target/foreign system must grant access. Read/write access can be restricted by setting zero length for segment. AIM (capability) permission must be less or equal. [Hong82] Hong, T.H., Shneier, M., Hartley, R. and Rosenfeld, A. "Using pyramids to detect good continuation", University of Maryland, Computer Science TR 1185, 1982 Cited in Lipscomb91 [Hopfield83] Hopfield, et al "Unlearning Has a Stabilizing Effect in Collective Memories", Nature, Vol 304 pp , 1983 Cited in Faggin89 [Hosaka81] Hosaka, M. and Kimura, F. "Use of handwriting action in construction of models", in Scientific Information Systems in Japan, H. Inose, editor, 1981, pp Tappert's bibliography [Hosaka82] Hosaka, M. and Kimura, F. "Using Handwriting Action to Construct Models of Engineering Objects", Computer, Vol 15 No 11, November 1982, pp User interface (fill in charts) for handwriting recognition input of engineering drawings (mechanical drafting, maps, NC control drawings) Features are passage through 3x3 template grid, then chord lengths and directions (cusp, stroke, rotation, straight): cusps turn into small loops, vice versa is a source of error [HowbrookE83a] Howbrook, E. "Apparatus and Methods for Recognizing Handwritten Signs", United States Patent 4,369,431, January 18, 1983, assigned to National Research Development Corporation, London, England Signature recognition/verification using first twelve initial segments, segmenting by zero velocity in Y Notes that a frequency cut-off of 20Hz is o.k. for signature signal [Hsu82] Hsu, W.S., Takahashi, K., Ozawa, S., and Fujita, H. "Ordered stroke extraction method for printed Chinese character recognition", Trans. of IECE of Japan, Vol E65 No 2, February 1982, p. 140 (abstract only) NTIS abstract: fix skeletonization/line-thinning breakdown at intersections by simulating writing motion in OCR [Huber83] Humber, William A. "Interactive Map Information Exchange System", United States Patent, 4,420,682, December 13, 1983 Optical scanner which read digital co-ordinates from a map, using co-ordinates in magnetic ink, but also describes optical scanning [Huh82] Huh, Y.K. and Beus, H.L. "On-line recognition of hand-printed Korean characters", Pattern Recognition, Vol 15, 1982, pp Tappert's bibliography [HullJJ82a] Hull, Jonathan J. and Srihari, S.N. "Experiments in text recognition with binary n-gram and viterbi algorithms", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-4, 1982, pp Viterbi and binary n-gram for context in optical character recognition: most efficient implementation: Viterbi algorithm is based on probabilities of confusion of pairs of characters [HullJJ82b] Hull, Jonathan J. and Srihari, S.N. "Comparison of two contextual post-processing algorithms for text recognition", Proceedings of 1982 IEEE Computer Society Conference on Pattern Recognition and Image Processing, 1982, pp (abstract only) NTIS abstract: looks very similar to other 1982 paper by same authors. [HullJJ83b] Hull, Jonathan J., Srihari, S.N., and Choudhari, R. "An integrated algorithm for text recognition: comparison with a cascaded algorithm", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-5, 1983, pp Letter substitution errors on text recognition corrected by various means of context: combining bottom-up and top-down (syntactic and semantic) context works better than separately Context for spelling correction: goodness measure extended to probability that it is a corrupted form of another letter Context for spelling correction: letter probabilities: digrams/pairs and trigrams (bottom/up context) (may involve a false assumption about input language) Context for spelling correction: dictionary lookup (lexicon for top/down context) Many sources for spelling errors: typographical in original text, keying/writing errors, character recognition error [HullsLR83a] Hulls, L. Robin "On-Board Intelligence Increases Accuracy of Plotters and Digitizers", Computer Technology Review, Summer 1983, pp :26:20 PM]

84 ( ) History of Pen and Gesture Computing Numonics article on digitizers and stepper-motor plotters Digitizers: no point being more accurate than the application needs. accuracy may be affected by pen angle/tilt. two-phase coarse/fine position determination using two characteristics (interpolation?) [IBM81a] IBM Technical Disclosure Bulletin "Improved Parameter Set for Adaptive Symbol Recognition", IBM Technical Disclosure Bulletin, June 1981, pp Copy on file does not give name of author: since it mentions "elastic matching", most likely it is C.A. Tappert Use offset of character from its center of gravity as a substitute for the writing baseline Add horizontal positions of strokes to each other as an additional factor in recognition: computer with Functional Attributes of Shillman/Blesser (example is "A" and three-stroke "asterisk") Also filed under Tappert and under Karnaugh [IBM81b] IBM "Liquid Crystal Display and Touch Panel Keyboard Input", IBM Technical Disclosure Bulletin, September 1981, pp Cited in More93 [IEEE83a] IEEE CG&A "New Products: Software brings CAD capability to microcomputers", IEEE Computer Graphics and Applications, April 1983 Product brief on AutoCad: light-pen, or Sun-Flex touch pen (touchscreen digitizer) [Ikeda81] Ikeda, K. et al "On-line Recognition of Hand-Written Characters Utilizing Positional and Stroke Vector Sequences", Pattern Recognition, Vol 13 No 3, pp , 1981 Discrimination of similar characters: boundary recognition. Uses different methods for characters of different number of strokes [ImageData82] Image Data Products "Image Data Tablet System product information", Bristol, England, 1982 Digitizer tablet [Inforite82] Cadre Systems Limited "Inforite Hand Character Recognition Terminal product information", Cadre Systems Limited, 1 Wilkinson Road, Cirencester, Glos., GL7 1YT, England Early British handwriting recognition product: single line display, paper forms (shop invoices) fit into device, shown at Comdex 1982 Note: Inforite used as product name by other companies [IshiiK83a] Ishii, K. "Generation of Distorted Characters and Its Applications", Denshi Tsushin Gakkai Ronbunshi, Vol 66-D No 11, November 1983, pp (in Japanese), translated in Systems, Computers and Controls, Vol 14 No 6, 1983, pp Cites problem of needing very large training samples to get all variations Claims "only samples of low quality can improve dictionary" (training) (vs. ambiguous?) Claims "recognition rate tells you nothing about how performance goes down with quality of characters" Shows artificial forms used in Japanese JIS handprint standard Not enough to read good characters, but how well does it do on bad ones? Claims 98.5% recognition rate Use artificial variability instead of real handwritten data (!) [JainR82a] Jain, R. and Haynes, S. "Imprecision in Computer Vision", Computer, Vol 15 No 8, August 1982, pp Goodale83. Application of fuzzy set theory to visual recognition / OCR: imperfect lighting, other imprecise measurements. Variation in connected regions, thinning / edge detection. [JohnsonEO81] Johnson, E.O. and Tosima, S. "Visual-Perception-Related Effects in Chinese-Japanese Written Characters", RCA Review, Vol 40, March 1981, pp 60-ff (partial copy) Human recognition features: visual groups of strokes in Chinese/Japanese characters consist of subgroups of strokes, usually about four: visual group count corresponds to letter count in Western languages [JordanDM83a] Jordan, David M. "Multics Data Security", Scientific Honeyweller vol 2 no 1, June Derived version 1983, Honeywell document GA01 Describes finer control and finer parsing of access control in Multics with AIM Access Isolation Mechanism (nondiscretionary access control) in stricter set of rules for matches between segment and user attributes. AIM is to prevent user from releasing data to which user may have access via ACL: User's access is the more restricted of the two: clearance (of user) vs. classification level (of data). [Kaleyeh83] Kaleyeh, J.M. and Landgrebe, D.A. "Predicting the Required Number of Training Samples", IEEE Trans. Pat. Anal. and Mach. International, November 1983, pp Cited in Meisel85 [Kamran83] Kamram, A. and Feldman, M.B. "Graphics Programming Independent of Interaction Techniques and Styles", Computer Graphics, Vol 17 No 1, January 1983, pp Describes GMU's Information Display Systems project (see Foley). Critique of CORE (and GKS) input device model, lack of extensibility [Karnaugh81] Karnaugh, M., Kurtzberg, J.M., and Tappert, C.C. "Improved Parameter Set for Adaptive Symbol Recognition", IBM Technical Disclosure Bulletin, Vol. 24, No. 1B, pp , 1981 Cited by Marlin Eller, Microsoft Pen Computing group available at / ip.com [Kato82] Kato, O., Iwase, H., Yoshida, M., and Tanahashi, J. "Interactive Handdrawn Diagram Input System", Proceedings of IEEE Computer Society Conference on Pattern Recognition and Image Processing, June, 8:26:20 PM]

85 ( ) History of Pen and Gesture Computing 1982, Las Vegas, Nevada, pp Interactive user-interface for handwriting recognition, sketching and sketching, using angle variation and stroke type (straight line, angled line, ellipse, circle, arc) for features Contains user interface for text entry to pretty up character spacing and alignment, fixing sketches (without recognition) [KimJ83a] Kim, J. "Baseline Drift Correction of Handwritten Text", IBM Technical Disclosure Bulletin, Vol 25 No 10, March 1983, pp See also Tappert papers Correct cursive script for baseline drift [KimS81] Kim, S. "Inversions - a catalog of calligraphic cartwheels", BYTE Books, Peterborough, New Hampshire, 1981 Very large catalog of reversible and ambiguous text to show variability of human recognition and perception. Has informal but very interesting bibliography on variability of human recognition [Kirsch82] Kirsch, Steven T. "Electro-Optical Mouse", United States Patent 4,364,035 Optical mouse of surface having passive, position-related marks in a pattern of two colors / Sekendur? (Shading optical tablet) [Kirsch83] Kirsch, Steven T. "Electronic Mouse", United States Patent 4,390,873, June 28, 1983 Optical mouse using checkerboard square pattern on the mouse-pad / tablet: Sekendur? [KnoxKT82a] Knox, Keith T. "Image Processing Method and Apparatus Having a Digital Airbrush for Touch Up", United States Patent 4,345,313, assigned to assigned to Xerox Corporation, Stamford, Connecticut, August 17, 1982 For brush-type user-interface in graphics drawing [Krouse83] Krouse, John K. "Selecting a Graphic-Input Device for Cad/Cam", Machine Design, October 6, 1983, pp Voice/speech/tablet/mouse/joystick input overview, 1983 [KrummeDW82a] Krumme, D.W., and Ackley, D.H. "A Practical Method for Code Generation Based on Exhaustive Search", Proceedings of SIGPLAN '82 Symposium on Compiler Construction, A.C.M. SIGPLAN Notices, Vol 17 No 6, June 1982 Fortran compilers beat the heck out of "C" compilers for code efficiency [KruskalJB83a] Kruskal, Joseph B. "An Overview of Sequence Comparison: Time Warps, String Edits, and Macromolecules", SIAM Review, Society for Industrial and Applied Mathematics, Vol 25 No 2, April 1983, pp Distance metric for differences in linear strings: Levenshtein distance: mutations in chromosomes, UNIX "diff" files, string matching, minimal mutation distance etc. Clustering /boundary definition using relative difference (dyadic), not absolute position (monadic) Levenshtein distance easier to use than probabilistic/statistical estimate of how much change would be required Common sense: if your pattern recognition algorithm works better, it is better Describes boundary comparison using monadic variables/features (absolute value) vs dyadic (relative comparison only) Different approaches for dyadic comparison: common sense, adapt monadic variables, and Levenshtein relative distance [KuhlFP82a] Kuhl, Frank P. and Giardina, Charles R. "Elliptic Fourier Features of a Closed Contour", Computer Graphics and Image Processing Vol 18, pp , 1982 Recognize object outlines by Fourier analysis of 2D chain-code outline, lowest mode of fit is an ellipse. Mentions quantization / pixelation of images, show effects. [KuklinskiT82a] Kuklinski, T. and G. Babb "Pattern Algorithm Permits Freehand Printed-Data Entry", Computer Technology Review, Winter 1982 Pencept pendpad terminal product, handwriting recognition on tablet. [Kurtzberg82] Kurtzberg, J.M. and Tappert, C.C. "Segmentation Procedure for Handwritten Symbols and Words", IBM Technical Disclosure Bulletin, Vol 25 No 7B, December 1982, pp Mentions reduction of "dots" as a stroke type Makes disparaging remarks about "boxed" input for discrete recognition Discusses delayed strokes, "t"-crossings for discrete recognition [Kuzunuki83] Kuzunuki, Sochiro "Online Handwritten Input Graphic Editing Device (translation)", Japanese patent application JP , March 1983 (translation) Japanese patent showing segmentation, handwriting, sketch input, drawing extents Cited in Sklarew patents [Lai81] Lai, M.T.Y. and Suen, C.Y. "Automatic recognition of characters by Fourier descriptors and boundary line encodings", Pattern Recognition, Vol 14, 1981, pp Cited in Kerrick88, Li89 [Lambden81] Lambden, Martin R. "Electrographic Apparatus", United States Patent 4,289,925, September 15, 1982 Quest Automation MicroPad, handwriting recognition product, small stand-alone terminal: pressuresensitive tablet using flexible membrane under tension, resistive film [LampsonBW83a] Lampson, Butler W. "A Description of the Cedar Language: A Cedar Language Reference 8:26:20 PM]

86 ( ) History of Pen and Gesture Computing Manual", Xerox PARC technical report CSL-83-15, December 1983 Reference manual for Cedar object-based language, system, and GUI, successor to Smalltalk. Cedar kernel language is low-level subset, with direct access to primitive or OS types not permitted in Cedar. Strong type checking. No description of Cedar GUI. [LandauerTK83a] Landauer, T.K., Galotti, K.M. and Hartwell, S. "Natural Command Names and Initial Learning: A Study of Text-Editing Terms", CACM, Vol 26, July 1983, pp Cited in Rhyne86 Rhyne86 cites this that for verbal command names, very poor agreement on informal name subjects give for text editing commands (mnemonicity for gestures?) "naturalness" in user-interface not helpful: for example, better if different names for options with similar semantics, but different syntax Human factors / user-interface: different methods of eliciting preferred command names get different results for same subject -- in particular, naive users make poor choices for command names [LeeA83a] Lee, Allison and Lochovsky, F.H. "Enhancing the Usability of an Office Information System Through Direct Manipulation", Proceedings of the CHI 1983 Conference on Human Factors in Computing Systems, Boston, 1983, pp Cited in Kankaanpaa87 gesture/handwriting user interface on digitizing tablet (using puck/stylus). Text editing gestures for add (caret), delete (underline), replace (inverted pigtail), paragraph, change font, etc. Trash can, filer, other icons (compare with Wang Freestyle?). [LeeB81] Lee, Bum C., Kim, Jung G. and Yi, Seung, K. "Improvement on Korean Character Recognition by Resolving Ambiguity Problem", Proceedings of International Conference on Cybernetics and Society, IEEE 0360/8913/81/ , October, 1981, Atlanta, Georgia, pp Korean recognition: six types of Korean characters (first consonants, second consonants, vertical vowels, horizontal vowels) using syntactic recognition and 8-direction chain codes [Lemone82] Lemone, Karen A. "Similarity Measures Between Strings Extended to Sets of Strings", IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol PAMI-4, No 3, May 1982, pp substrings of chain-codes [Leroux81] Leroux, J., Miclet, L., Bonnet, A., Delarue, X. and Tormos, S. "Segments Detection in Binary Pictures for the Representation and the Syntactic Recognition of Hand Written Characters", Proceedings of the 6th International Conference on Pattern Recognition, IEEE CH1801-0/82/0000/ , pp Chain code string comparison in OCR for handwriting recognition? [Litvin82a] Litvin, Y. "Segmentation of Handwritten Text by the Analysis of Two-Element Connectors", unpublished manuscript, G.T.E. Research Laboratory, Waltham Massachusetts, 1981 Cited in Litvin82b [Litvin82b] Litvin, Y. "Principles of evaluation for hand-printed and cursive text recognition methods", G.T.E. Technical Note 401.1, April 1982 See also Recognitive Sciences and Skylight Software (Yuri's consulting business names). Mentions retrace removal (page 7) User interface: points out difference between errors understandable to user, and errors not ("qualitative errors"). [Litvin82c] Litvin, Y. "Two Implementations of Data Reduction in Graphics Input", unpublished manuscript, December 1982 Cited in Goodale83 [LoomisJ83a] Loomis, Jeffrey; Poizner, Howard; Bellugi, Ursula; Blakemore, Alynn; and Hollerbach, John "Computer Graphic Modeling of American Sign Language", Computer Graphics, Vol 17 No 3, pp. 105-ff Uses LED lights and digital cameras as three-dimensional 3D digitizer sensor to recognize hand gestures of ASL American Sign Language [LoomisJM81a] Loomis, Jack M. "Tactile pattern perception", Perception, Vol 10, 1971, pp Assistive technology for the blind: compare with tactile/haptic Braille displays. Analysis of spatial bandwidth (resolution) limitations of human touch for tactile input. Cites to Optacon, TVSS vibrotactile display for the blind. [Loy82] Loy, W.W. and Landau, I.D. "An On-Line Procedure for Recognition of Handprinted Alphanumeric Characters", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-4 No 4, July 1982, pp Reduce handwritten characters to a polygon / Freeman chain codes, then compare syntactic feature vector, then statistics on segment lengths Serif/hook removal, preprocessing/smoothing, retrace collapsing on on-line character recognition Handwritten samples "only" constrained to one of 69 writing styles/shapes: 99% and 97%: 20Kbytes memory, 500 Ms on 8086 Asserts for on-line recognition that learning new shapes (adapting to different user's writing?) is more important than accuracy / recognition rate [LukisLJ82a] Lukis, Larry Joseph and Duhig, Gerard Peter "Writing pad for a character recognition device", European Patent Application EP A1, June 23, 1982 Resistive sheet tablet, spacer dots (compare Elographics). Hand rejection / resting hand does not press hard enough (compared to stylus point) to make contact between the sheets. 8:26:20 PM]

87 ( ) History of Pen and Gesture Computing [MallosJB82a] Mallos, James B. "Touch Position Sensitive Surface", United States Patent 4,346,376, August 24, 198 Touchscreen over CRT display using total internal reflection of light generated by CRT, reflected where finger is in contact with screen. Like a lightpen, detection of position by knowing timing of CRT raster position, therefor unaffected by CRT raster drift. [Mantas83] Mantas, J. and Heaton, A.G. "Handwritten character recognition by parallel labelling and shape analysis", Pattern Recognition Letters, Vol 1, July 1983, pp Refers to problem of encountering a shape system not trained to for adaptive recognition OCR of handwriting recognition using polygonal approximation (chain codes), fuzzy labelling: thinning, tail-removal, fuzzy sets. [Marr82] Marr, D. "Vision: A computational investigation into the human representation and processing of visual information", San Francisco, Freeman Press, 1982 Cited in IchikawaS84. Human recognition of figures by "wire-frame" model. [MartinC81a] Martin C.; Jelinsky, P.; Lampton, M.; Malina, R.F.; and Anger, H.O. "Wedge-and-strip anodes for centroid-finding position-sensitive photon and particle detectors", Review of Scientific Instruments, Vol 52, 1981, p ff Charge-ratio electrostatic configuration for particle detection: resolution not limited by dimensions of wedge anodes and different-width strip anodes. Different geometries: dual wedge anodes between strip anodes, wedges in opposite directions of taper; radial configuration; zig-zag pattern with no vias / through holes in substrate. [Matsuda83] Matsuda, Ryouchi "Present Status and Future Trends of Japanese Language Information Processing Systems", Proc. of 1983 International Conference on Text Processing with a Large Character Set, Tokyo, Japan, October 17-19, 1983, pp Survey of problems and technologies for processing Japanese Characters: standardized keyboards, speech recognition, handwriting recognition [Matsukawa83] Matsukawa, Junko "Naming and recognition of random shapes", Japanese Journal of Psychology, Vol 54 No 1, 1983, pp (In Japanese) Humans recognize (recall? identify?) shapes better if the are recognizable as a familiar object [Maurer82] Maurer, H.A., Rozenberg, G., and Welzl, E. "Using String Languages to Describe Picture Languages", Information and Control, Vol 54, 1982, pp (partial copy) Chain codes: something similar to BLRTs for describing images [McDermott83] McDermott, Drew "Contexts and Data Dependencies: A Synthesis", IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol 5, No 3, May 1983, pp General paper on information dependencies and context [McDonaldWW82a] McDonald, Wylie W. "System and method for providing an audible sound and a tactile feedback in an electronic data processing system", United States Patent 4,334,280, June 8, 1982 Give a beep periodically (one second intervals) while a device is operating, or give an audible and tactile feedback each time a key is pressed / device is operated. For calculators, beeps on the keys? [Meads83] Meads, Jon A. "Defining the Ergonomic Buzzwords", Proceedings of the 1983 Annual conference of the A.C.M What is user-friendly? friendly to a beginner may be bad for expert [Meguro82] Meguro, S. and Umeda, M. "Recognition of multi-font Chinese Characters", Trans. Inst. Electronic and Communications Engineers, Japanese Section, Vol E65, No 8, p 513, August 1982 Multi-font OCR off-line recognition [MehtaN82a] Mehta, Nimish "A Flexible Machine Interface", M.A.Sc. Thesis, Dept. of E.E., U. Toronto, 1982 Early multi-touch input device, optical detection of fingers on frosted glass, but no rear-projection (separate display). Also recognize shapes and objects on surface. Cited by Buxton as historical note. [MehtaN82b] Mehta, Nimish; Smith, Kenneth C.; and Holmes, F.E. "Feature extraction as a tool for computer input", IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP '82. May 1982, pp multi-touch/multi-hand input tablet (camera underneath, optical) with projection of keyboard or template images on top. Reconfigurable keyboard/touch service. Compare with Kaplow (reconfigurable keyboard) and Jeff Han (FTIR multi-touch) and Wellner DigitalDesk. [Meyrowitz82a] Meyrowitz, Normand and van Dam, Andries "Interactive Editing Systems: Part I", Computing Surveys, Vol 14 No 3, September 1982 Text editing: primarily keyboard-based editing systems, long treatment of Xerox STAR with mouse pop-up menus, but no use of gesture (other than drag?), digitizer, stylus, or handwriting editing. [Meyrowitz82b] Meyrowitz, Normand and van Dam, Andries "Interactive Editing Systems: Part II", Computing Surveys, Vol 14 No 4, 1982 Text editing: primarily keyboard-based editing systems, long treatment of Xerox STAR with mouse pop-up 8:26:20 PM]

88 ( ) History of Pen and Gesture Computing menus, but no use of gesture (other than drag?), digitizer, stylus, or handwriting editing. [Micropad82] (no author) "Micropad Product Information", Quest International, American Sales Office for Micropad Inc., LaGrange Illinois handwriting-terminal using a digitizer with two conductive sheets, sheets covered by a hand-rest; singlecharacter discrete character recognition also showed a GUI application picking from a diagram of replacements parts, plus character recognition Press release included, data-entry use in New Scotland Yard: "gets write to the point"; Joe Crivello, National Sales Manager, Illinois [MoogRA82a] Moog, Robert A. "A Multiply Touch-sensitive Clavier for Computer Music Systems", Proc. International Computer Music Conference, 1982, pp Musical touch keyboard that detects X and Y position of fingers, key velocity (via position sensor), and force/pressure capacitive sensing. Z/Force detected with conductive rubber with greater capacitance as it is conformed by force to a curved bar capacitor plate, four-wire resistive sheet touchpad for X and Y. [MoranT81a] Moran, T. "The Command Language Grammar: A Representation for the User Interface of Interactive Computer Systems", International Journal of Man-Machine Studies, Vol 15, 1971, pp 3-50 (abstract only) Grammar and semantics for user-interfaces (commands). Example input devices are keyboard and display. [Morasso83] Morasso, P., Mussa Ivaldi, F.A., and Ruggiero, C. "How a Discontinuous Mechanism can Produce Continuous Patterns in Trajectory Formation and Handwriting", Acta Psychologica, Vol 54, 1983, pp Generative variability in handwriting [Mullin81] Mullin, James K. "Reliable Indexing Using Unreliable Recognition Devices", IEEE Trans on Pattern Analysis and Machine Intelligence, Vol PAMI-3, No 3, May 1981, pp Indexing document automatically with OCR recognition by substituting likely-confused characters to same pseudo-character, similar to the Soundex system by Davidson for similar-sounding English names. Electronic ink / sketch indexing? [Murase83] Murase, H., Wakahara, T., and Umeda, M. "Online Recognition Algorithm for Hand-Sketched Flowchart by Candidate Lattice Method", Denshi Tsushin Gakkai Ronbunshi, Vol 65-D No 6, June 1983, pp (in Japanese), translated in Systems, Computers and Controls, Vol 14 No 3, 1983, pp Claims 97.9% recognition rate on 120 samples (small sample size) Refers to segmentation errors in recognizing two-dimensional flowchart symbols Sketch recognition - flowcharts [MurrayJM82a] Murray, John M. and Klingenstein, Kenneth J. "The Architecture of an Electronic Book", IEEE Trans. Industrial Electronics, Vol. IE-29 No 1, February 1982, pp Estimate of hardware requirements (memory, processor, display) for an electronic book including graphics, with search and indexing functions. Reference works (PDR, etc.), educational textbooks, coupled with speech synthesizers for the blind. Exemplary UI would be keyboard/buttons. [MyersCS81a] Myers, C.S. and Rabiner, Lawrence R. "Connected digit recognition using a level building DTW algorithm", IEEE Trans. Acoustics, Speech, and Signal Processing, Vol ASSP-29, 1981, pp Cited in Kruskal83: Dynamic time warping for recognition [NEC82] NEC: "Terminal that accepts handwriting lets the uninitiated use computer", Electronics Magazine, June 30, 1982, p 76 NEC handwriting terminal product for personal PCs, low-cost [NEC83] NEC "News Update: NEC 2100", Electronics Magazine, June 16, 1983, page 32 NEC 2100 kanji and hiragana symbols... product NEC handwriting terminal product, high-end version of personal PC product [NTT81] NTT "NTT 1900: System reads kanji characters into word processors", Electronics Magazine, June 16, 1981, page 64 NTT 1900 kanji and hiragana symbols product "Aesop" on-line handwriting recognition, Nippon Telephone and Telegraph See also list in CIC folder on NTT [NTT82] NTT: "System edits handwritten copy, finishes sketches", Electronics Magazine, June 30, 1982, pp (Date may be wrong) sketch/scribble/gesture input and editing system. Mechanical digitizer(!). [Nagura83] Nagura, Masakazu and Suenaga, Yasuhito "A Facsimile-Based Graphics Editing System by Auxiliary Mark Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-5 No 4, July 1983, pp See also Suenaga80, same paper Handwriting mark recognition, OFF-LINE scanned OCR user interface for graphics editing Mark-up OFF-LINE scanned handwriting recognition user interface for on-line changes to scanned line drawings Has many Japanese citations for OCR of handwritten drawing and character recognition [NagyG82a] Nagy, G. "Optical Character Recognition - Theory and Practice", in "Handbook of Statistics", Vol 2, Krishnaiah, P.R. and Kanal, L.N., editors, North-Holland, 1974, pp Testing: substitution error rates for OCR in practice two to three orders of magnitude lower than reported in academic literature Optical digitizer resolution of 0.004" sufficient for typed text Human adaptation 8:26:20 PM]

89 ( ) History of Pen and Gesture Computing reasons for some handwriting recognition systems' success Cursive writing not as useful as speech, or discrete writing Most optical scanners for OCR barely have resolution adequate for recognizing ideal characters, much less real ones Optical scanning digitizer characteristics: geometric, photo-metric, control Optical scanning digitizer characteristics: cite for tablet digitizer as comparison Testing: cites work by Chow on statistical relation of substitution vs reject error rate Kahan87 cites this as saying Duda72 binary Bayesian statistical classifier is widely used in OCR Optical digitizer characteristics: no vendor willing to be pinned down on performance [NagyG83a] Nagy, G. "Optical Scanning Digitizers", IEEE Computer, May 1983, pp Optical scanning digitizer characteristics: geometric linearity Optical scanning digitizer characteristics: stability/repeatability Optical scanning digitizer characteristics: cite for tablet digitizer as comparison [NagyG83b] Nagy, G. "Candide's Practical Principles of Experimental Pattern Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-5 No 2, March 1983, pp Tongue-in-check list of how researchers fudge pattern recognition statistics/results to bias them in their favor [Nakajima81] Nakajima, K., Kida, H., and Arakawa, H. "Handprinted Character Recognition Techniques on Commercial-Based Facsimile Input", Electrical Communications Laboratories Technical Journal, Vol 30 No 9, 1981, pp (in Japanese) (abstract only) Handwritten numerals and katakana recognition, coping with facsimile distortions and variability [Nakamura83] Nakamura, Y. "Character Reading Apparatus", United States Patent 4,389,634, June 21, 1983, assigned to Tokyo Shibaura Denki Kabushiki Kaisha, Kawasaki, Japan, and Tokyo Electric Company, Limited, Tokyo, Japan Hand-held scanner for character recognition [NakataniLH83a] Nakatani, Lloyd H. and Rohrlich, John A. "Soft machines: A philosophy of user-computer interface design", Proc. CHI '83, pp Simulate real machines / controls for devices using real-time computer graphics display and a touch screen digitizer: refers to switches and controls appearing on display in response to input from other switches or controls, such as a calculator changing modes and keys. Compre with Kaplow? [NelsonBJ81a] Nelson, Bruce Jay "Remote Procedure Call", XEROX PARC Report CSL-81-9, May 1981: Ph.D. dissertation CMU report CMU-CS Generally credit with coining term RPC Remote Procedure Call. RPCs integrated into computer, or preprocessor for source file. Marshalling, exceptions, stub functions. Cites to other references for PKI authentication, authorization mechanisms. See also RFC 707, [Newbower81] Newbower, R.S.; Cooper, J.B.; Edmondson, J.E.; and Maier, W. Reynolds "Graphics-tablet for Data Entry in Computer-assisted Anesthesia Record-keeping",?? Conference Proceedings, IEEE /81/0000/0139, 1981, pp User-interface application: special symbols and markings with handwriting recognition for application involving anesthesia record keeping User-interface: shows forms with combinations of writing, handwriting recognition, drawing, check-off menu areas, etc. [Nihei83] Nihei, Y. "Developmental Change in Covert Principles for the Organization of Strokes in Drawing and Handwriting", Acta Psychologica, Vol 54, 1983, pp Change in writing styles as kids grow up [Numonics82] Numonics "DigiBit Product Description", Numonics Incorporated, 418 Pierce Street, Lansdale PA 19446, 1982 Vendor of electromagnetic digitizer tablets: small vendor, mostly custom products. [OCR81a] (various) "OCR/scanner products circa 1981", (various) Folder of OCR products circa 1981: Burroughs 1200 and Burroughs 1205 Series, AEG Formularleser PFL6160 POLYFONT, ECRM Concept 1 Pagereader; Burroughs PS100; DEST Corporation WorkLess Station; Datapro Research "All About Optical Readers" 1978; Datacopy CIR Software Character Image Recognition; Kurzweil KDEM 1200 Intelligent Character Recognition; Compuscan Alphaword Series 80 [Odaka81a] Odaka, Kazumi, Wakahara, Toru, and Hashimoto, Shin'ichiro "Online Handwritten Character Recognizer - An Application to Japanese Word Processor", EC Vol 81 No 20, pp (in Japanese) Japanese handwriting recognition / text editing, boxed (large format) input on tablet integrated into desk surface. [Odaka81b] Odaka, K. "On-line Pattern Recognition System for Hand-written Characters", United States Patent 4,284,975, August 18, 1981, assigned to Nippon Telegraph and Telephone Public Corporation, Tokyo, Japan NTT character recognition patent: Use of "feature points" in character recognition vs octants, etc. [Odaka82a] Odaka, K. and Masuda, I. "Pattern Recognition System for Hand-written Characters Operating on an On-Line Real-Time Basis", United States Patent 4,317,109, February 23, 1982, assigned to Nippon Telegraph and Telephone Public Corporation, Tokyo, Japan NTT character recognition patent Patent on stroke-order independent recognition for Kanji/Chinese Feature 8:26:20 PM]

90 ( ) History of Pen and Gesture Computing points for Kanji/Kana are lengths of segments [Odaka82b] Odaka, Kzaumi, Arakawa, Hiroki and Masuda, Isao "On-line Recognition of Handwritten Characters by Approximating Each Stroke with Several Points", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-12 No 6, November 1982, pp See 1980 paper of same title in Japanese Better than 99.8% correct recognition on Chinese/Kanji, hiragana, katakana, and alphanumerics/romanji. Three to six feature points on each stroke for on-line handwriting recognition: points just resampled to a minimum distance? end-points of (Chinese) straight strokes only? [Odaka82c] Odaka, K.; Wakahara, T.; and Masuda, I. "Stroke Order Free On-line Character Recognition Algorithm and Its Application", Trans. Inst. Electonics and Commun. Eng. of Japan, Vol E65 no 6, June 1982, pp. 379-ff (partial copy) Shape matching for handwriting recognition, followed by specific algorithms for ambiguous cases. Uses distance measurements and not stroke-order information. [Odaka83a] Odaka, K., Wakahara, T., Masuda, I., and Hashimoto, S. "Stroke Order Free Online Character Recognition Algorithm and Its Application", Electronic Communications Laboratory Technical Journal, Vol 32 No 10, 1983, pp (in Japanese) Japanese handwriting recognition: features are inter-stroke distance pairs: claims 99.5% accuracy on 2057 Kanji Refers to AESOP user-interface for handwriting text/script editing. Measures accumulated distance along each stroke of an input with the strokes of an ideal sample -- compare with Tappert elastic matching? [Ogawa81] Ogawa, H. and Taniguchi, K. "Stroke resolution and segmentation of character string based on relaxation techniques", Trans. of IECE Japan, Vol PRL80-5, 1981 Cited in Leung87 [OkaR83] Oka, R. "Studies on Recognition of Handwritten Chinese-Japanese Characters by Using Cellular Features", Research of the Electrotechnical Laboratories, Report No , September 1973 (in Japanese) Cellular automaton for character recognition (?) [Okamoto83] Okamoto, N., Nakamura, O., and Minami, T. "Character Segmentation for Mixed-Mode Communication", Proceedings of IFIP 9th World Computer Congress, September, 1983, Paris France, pp Character and word segmentation in OCR documents using overlapping rectangles [Okamura83] Okamura, K., Morita, K., Kanaoka, T., Okada, T., and Tomita, S. "Syntactic Pattern Recognition for Handwritten Katakana Characters by a Bottom-up Parser", Trans. of the Institute of Electronics and Communications Engineers of Japan, Vol J66D No 2, February 1983, pp (Abstract only) NTIS abstract of Japanese publication: Katakana Japanese handwriting recognition [Pavlidis82a] Pavlidis, Theo "An asynchronous thinning algorithm", Computer Graphics Image Processing, Vol 20, 1982, pp Cited in Arcelli85 Arcelli85 cites as general reference on line thinning, and its applications [Pavlidis82b] Pavlidis, Theo "Algorithms for Graphics and Image Processing", Berlin, West Germany: Springer- Verlag, 1982 Cited in Arcelli85 Arcelli85 cites chapter 9 as general reference on line thinning, and its applications [Pavlidis83a] Pavlidis, Theo "Effects of Distortions on the Recognition Rate of a Structural OCR System", Proceedings of Computer Vision and Pattern Recognition '83, Washington, D.C., June 1983, pp Cited in Lam86 3-point calibration on digitizer tablets (not four points) [Pencept82a] Pencept "Hand Print Recognition Technology Provides A New Way To Use Computers", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1982 handwriting-input terminal product: DCR dynamic character recognition (a.k.a. on-line handwriting recognition). Mentions boxed input for stroke parsing/segmentation. Tablet 11" high by 15" wide: standard paper with printed form on left, control area on right. [Pencept83a] Pencept "Pencept Penpad (TM) 200 Product Literature", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1983 handwriting-input terminal product [Pencept83b] Pencept "PENPAD Reference Manual", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1983 (hardcopy manual) Handwriting terminal with digitizing tablet and handwriting recognition Handwriting recognition terminal from Pencept: VT-100 terminal emulation, with internal language for forms management. P. 3-7 keyboard emulation mode for input. p virtual tablets, virtual areas with independent programming, coordinate origin/rotation/scaling. [PepperW81a] Pepper, William Jr. "Touch panel system and method", United States Patent 4,293,734, October 6, 1981 Resistive-film touchpanel, finger may either be source or sink of signal. Signal flowing through user's body 8:26:20 PM]

91 ( ) History of Pen and Gesture Computing to ground, or signals generated by ambient noise (e.g. 60-Hz hum). Pressure-sensitive touch panel, transparent touch panel using ITO or transparent polyester film. Triangular resistive sheet touchpanel with three sides. [PepperW81b] Pepper, William Jr. "Video game apparatus and method", United States Patent 4,302,011, November 24, 1981 Touch-screen digitizer that senses intensity of touch force or pressure [PepperW82] Pepper, William Jr. "Touch panel system and method", United States Patent 4,353,552, October 12, 1982 Resistive-film touchpanel, finger may either be source or sink of signal. Signal flowing through user's body to ground, or signals generated by ambient noise (e.g. 60-Hz hum). Pressure-sensitive touch panel, transparent touch panel using ITO or transparent polyester film. Triangular resistive sheet touchpanel with three sides. Displays icons for touch locations (indicia). [PepperW83a] Pepper, William Jr. "Edge terminations for impedance planes", United States Patent 4,371,746, February 1, 1983 Edge connectors to linearized distortions at edges of resistive film digitizer / touch-screen [PetersC83a] Peters, Chris "The Microsoft Mouse: Letting the mouse out of the bag", BYTE Magazine, June 1983, pp Microsoft mouse: article compares mouse to light pen and to joystick. Piano demonstration program. [PetersenP83a] Petersen, Peter "Man-Machine Communication: CERT Thesis", PhD Thesis, CERT / Aalborg Universitetscenter, 11 January 1983 Review of CERN user-interface (man-machine interface) developments, including capacitive touchscreen. Capacitance measurement by charge-time detector (did not work well), and by voltage divider (worked better). 16-button touchscreen with hierarchical menus. Cites to touchscreens based on cross wires, infrared light, acoustic waves, resistive, capacitive. [PickHL83a] Pick, Herbert L. Jr., and Teulings, Hans-Leo "Geometric Transformations of Handwriting as a Function of Instruction and Feedback", Acta Psychologica, Vol 54, 1983, pp Tell users to change their handwriting: changing slant works, but if say to make taller, users also make wider, and vice versa. (Handwriting variability) [Plamondon83] Plamondon, R. and Brault, J-J. "A System for Signature Analysis and Verification Based on an Accelerometer Pen", Proceedings of International Carnahan Conference on Security Technology, Zurich, Switzerland October 4-6, 1983, pp Uses tilt angle in signature verification [PopularScience81a] Popular Science "Electronic printer transmits mail, makes copies, too", Popular Science, February 1981 Xerox 5700 printer with touch-screen interface showing simulated buttons and switches, touch-to-begin [Quest82] Quest Automation Limited "Micropad User's Guide", Quest House, Prince's Road, Ferndown, Dorset BH22 9HQ, United Kingdom Micropad dynamic on-line character recognition product: Resistive film digitizer? [Quest83] Quest Automation Limited "Q-Sign Terminal product literature", 10 Whittle Road, Wimborne, Dorset BH21 7SD, United Kingdom. Tel: , 1983 Quest Automation: Micropad signature verification product [Ray81] Ray, A.K. and Chatterjee, B. "An Algorithm for the Recognition of Constrained Handwritten English Numerical Characters", Journal of the Institute of Electronics and Telecommunications Engineers of India, Vol 27 No 9, September 1981, pp NTIS abstract: OCR on handwriting recognition of numerals: features are true endpoints, true group points, and true cross points [RecycledGoods83a] Recycled Goods, Inc. "Pencept M200 Penpad Tablet Computer - Vintage Collectible", fetched Dec 2007 Pencept Penpad M200 handwriting terminal, circa 1983, showing system box, digitizer, etc. [Rediffusion82] Rediffusion Computers Limited "WRITAWAY product literature", Kelvin Way, Crawley, Sussex RH10 2LY, England, 1982 Micropad-like handwriting recognition product from England, using two resistive sheets and an air separator on the tablet. [Reilly81] Reilly, Douglas L., Cooper, Leon N., and Elbau, Charles "A Neural Model for Category Learning", Center for Neural Science and Department of Physics, Brown University, Rhode Island date approximate Neural (net) model for supervised learning, later applied to handwriting recognition [RivestRL83a] Rivest, Ronald L.; Shamir, Adi; and Adleman, Leonard M. "Cryptographic Communications System and Method", United States Patent 4,405,829, September 20, 1983 Patent on original RSA public-key encryption/decryption algorithm. [RocheleauRT81a] Rocheleau, R.T. "Coarse Position Digitizer", United States Patent 4,242,843, January 6, 1981, 8:26:20 PM]

92 ( ) History of Pen and Gesture Computing assigned to Summagraphics Corporation, Fairfield, Connecticut Digitizer with loop as the transmitter, grid as the receiver [Romein81] Romein, J.J. "Acoustic Writing Combination, Comprising a Stylus with an Associated Writing Tablet", United States Patent 4,246,439, January 20, 1981, assigned to US Philips Corporation, New York, New York Acoustic digitizer design: two ultrasonic sound sources on the stylus, permitting tilt / parallax correction Accuracy: pencil lines are 0.005", Ink is 0.01" wide, visual acuity is " max [SAC82] SAC "Digitizer Terminology and Comparability", Science Accessories Corporation, 1982, Southport, Connecticut [Saghri81] Saghri, J.A. and Freeman, H. "Analysis of the precision of generalized chain codes for the representation of planar curves", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-3, September 1981, pp Line-segment approximation shows average quantization error a function of grid size, not angle resolution: gives formula for grid size vs matching accuracy [Sakoe82] Sakoe, Hiroaki "A Generalized Two-Level DP-Matching Algorithm for Continuous Speech Recognition", Trans. of the I.E.C.E. of Japan, Vol E65 No 11, November, 1982, pp Dynamic programming, time-skipping, dynamic time warping, for speech (author has also published on handwriting character recognition) [Salkeld82] Salkeld, Robert J. and Sklarew, Ralph C. "Closed Space Structures", United States Patent 4,318,517, March 9, 1982 Same Ralph Sklarew of Grid computer: structures in space the form of a ring around a planet or other orbiting body: cites Larry Niven "Ringworld" as prior art [Salter83] Salter, L. "Variability of Japanese Characters", internal report, Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, September 1983 Analysis (no user studies) of variability expected in Japanese handwriting recognition, based on what had been found in ASCII alphabetic handwriting and stylus dynamics [Samet81] Samet, Hanan "An Algorithm for Converting Rasters to Quadtrees", IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol PAMI-3, No 1, January 1981, pp 93-ff Image processing: group lines/regions in scanned raster image / bitmap into quadtrees, where tree nodes are maximal blocks (e.g. minimal fragmentation of regions) [SasakiL81a] Sasaki, L.; Fedorkow, G.; Buxton, W.; Retterath, C.; and Smith, K.C. "A Touch-Sensitive Input Device", Intl. Computer Music Conference Proceedings, 1981, pp Single-touch pressure sensitive (contact area) capacitive touchscreen/tablet for musical instrument input. Matrix of X and Y conductors, measure capacitance change on conductors individually. Adjust two values at once using X/Y input of tablet, into SSSP digital synthesizer music program. Third value from "pressure" as amount of capacitance change with surface area of contact. Multi-touch input would be better. Cited in Lee 85. [SatoT82] Sato, T. and Toja, A. "Recognition and Understanding of Hand-drawn diagrams", Proceedings of the 6th International Conference on Pattern Recognition, pp , 1982 Cleaning up graphical drawings using low-level symbol recognition: (see sketch editing, prettifying) [SatoY82] Sato, Yuichi and Nakamura, Taichi "Predictive Encoding Method for Handwriting Signals", Trans. of the IECE of Japan, Vol E 65 No 2, February 1982, p. 133 (abstract only) 30 Hz sampling of handwriting on a tablet, DPCM/PCM coders to send handwriting at bits/second, bit buffer: tablet performance/handwriting signal extraction/frequency response [Schaeken82] Schaeken, B. and Verschueren, W. "A Recognition System for Handwritten Numerals", Proceedings of the 6th International Conference on Pattern Recognition, Munich, Germany, October 19-22, 1982 NTIS: Supervised learning on statistical recognizer [SchmandtC81a] Schmandt, Christopher and Hulteen, Eric A. "The Intelligent Voice-Interactive Interface", Proc. CHI'82, pp Put-That-There project: hand-pointing gestures and voice command input. 3D gestures using Polhemus digitizer. [SchulzeLJJ83a] Schulze, Lawrence J.J. and Snyder, Harry L. "A Comparative Evaluation Of Five Touch Entry Devices", Virginia Polytechnic Institute report AD-A , October 1, 1983 Comparison of touchscreen digitizer / touch entry devices (TED) technologies: touch wire, cross wire, capacitive, conductive film (resistive), acoustic ranging, infrared (optical beam) and pressure sensitive. Cites to E.A. Johnson 1960's for invention of touch entry device. [Shillman81] Shillman, R.J. "Dynamic Character Recognition: An Emerging Technology", invited paper, Proceedings of COMPCON 81, 22nd IEEE Computer Society International Conference, IEEE Catalog No 81- CH1626-1, February 23-26, 1981 Handwriting input overview paper by Bob Shillman Gives more recent address for Micropad Limited, Image Data Products Limited, handwriting commercial vendors 8:26:20 PM]

93 ( ) History of Pen and Gesture Computing [ShinghalR82a] Shinghal, R. and Suen, C.Y. "A Method for Selecting Constrained Hand-Printed Character Shapes for Machine Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-4 No 1, pp Shows lots of collected base forms (with Suen) [ShinghalR82b] Shinghal, Rahan "An Experimental Investigation of Four Text Recognition Algorithms", IEEE Trans. System Man and Cybernetics, Vol SMC-12, No 4, July/August 1982, pp Context helps on character recognition: specifically modified Viterbi algorithm, even on "unconventional" English text. [ShinghalR83a] Shinghal, R. "A hybrid algorithm for contextual text recognition", Pattern Recognition, Vol 16, 1983, pp Context via Markov method, plus a dictionary, better than predictor-corrector method Cited in Sinha88: on spelling/dictionary context correction [ShiraiY82a] Shirai, Yoshiaki "Image Processing for Data Capture", IEEE Computer, November 1982 pp Review of extracting 3D information from 2D images: shading, binocular imaging: mostly machine vision. Also sketch recognition for hand-drawn (scanned) electronic circuit diagrams. [Shneiderman83] Shneiderman, B. "Direct Manipulation: A Step Beyond Programming Languages", IEEE Computer, Vol 16 No 8, pp 57-69, August 1983 Refers to IBM direct-manipulation office desktop user interface "Pictureworld" (like Microsoft's "Bob"?), with file cabinets, mailboxes, notebooks, phone messages Direct manipulation: gesture/command symbols user interface. States that direct manipulation UIs are more like actual objects people already comprehend, therefore easier to use (e.g. VisiCalc spreadsheet and a paper spreadsheet). Defines direct manipulation as manipulating digital objects on a screen without the use of command-line commands. Has definition of direct manipulation, page 64 [Shoukry83] Shoukry, Amin and Amin, Adnan "Topological and statistical analysis of line drawings", Pattern Recognition Letters, Vol 1, July 1983, pp On-line handwriting recognition using two-dimensional graph (chain codes) using slope/angle of lines, intersections, labyrinthology, for features [Siddiqui83] Siddiqui, K.J. and Shinghal, R. "Using Contextual Postprocessing to Improve Machine Recognition of Text", IEEE International Symposium on Information Theory, September, 1983, St. Jovite, Quebec (abstract only) Recognition performance 71% on Munsun's OCR handwriting data set, improved to 86% with Viterbi context algorithm [SinghB83] Singh, Baldev, Beatty, John C., Booth, Kellogg S., Ryman, Rhonda "A Graphics Editor for Benesh Movement Notation", Computer Graphics, Vol 17 No 3, pp 51-ff Digitizer tablet with four-button puck, experiment in GUI design with floating (pop-up?) and dynamic menus [SmithAR81] Smith, A. Richard and Erman, Lee D. "Noah -- A Bottom-Up Word Hypothesizer for Large-Vocabulary Speech Understanding Systems", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-3 No 1, January 1981, pp Context: using low-level speech context, hypothesizes word for higher-level context analysis. Context to resolve co-articulation on continuous speech recognition. Cites experiments to determine human use of context in speech. On speech recognition, discusses possible biases and justifications for them in training and test data collection. [SmithDC82a] Smith, David Canfield; Irby, Charles; Kimball, Ralpha; Verplank, Bill "Designing the Star User Interface", BYTE Magazine, 1982 No 4, pp Design of Xerox Star GUI with mouse and keyboard: mouse uses trackball rather than SRI wheels. Development philosophy: throw away first implementation, it was always a prototype. Properties and application sheets for graphical objects. Move is by click-source/click-move-button/click-target, not drag/move gesture. Keyboard windows / on-screen keyboard / virtuelle tastatur, similar to soft input panel SIP? [SmithDC82b] Smith, David Canfield; Irby, Charles; Kimball, Ralpha; Harslem, Eric "The Star User Interface: an overview", Proc. NCC AFIPS '82, June 7-10, 1982, pp Desktop user interface patterned after physical office: forms/records have no physical counterpart. Data icons and function icons, e.g. icons for printers, file cabinets, folders, property sheets. Keyboard windows / on-screen keyboard display, similar to soft input panel SIP? press-and-hold? Earlier references back to 1977 [SnellJM83a] Snell, John M. "Sensors for Playing Computer Music with Expression", Proc. Int'l. Computer Music Conf., 1983, pp Touch-sensitive touchpads in music for expression: add touch surfaces to clavier keyboard keys to send finger-force during a note, or movement in in Y direction (X is pitch, need the discrete keys). Back of black and white keys merge to a flat surface for continuous pitch control in X direction. Linear touch sensor may 8:26:20 PM]

94 ( ) History of Pen and Gesture Computing be sub-divided into sections for multiple fingers on same pitch (multi-touch). Reducing motion travel of key allows faster playing: substitute with force sensor. Cites multi-touch to Bill Buxton in 1983 for musical instrument input. Musicians rely on haptic feedback (feel of the physical instrument) in playing. Optical multi-touch input using "wine glass effect" of total internal reflection on a glass surface. [SnowberryK83a] Snowberry, K. et al "Computer Display Menus", Ergonomics, Vol 26 No 7, 1983, pp Hierarchical menu breadth / wide menus gives faster user actions than tall / narrow menus. [SpenceR82a] Spence, Robert and Apperley, Mark "Data base navigation: an office environment for the professional", Behavior and Information Technology, Vol 1 No 1, 1982, pp Paper study (as in paper-based prototype) for a interface for database use (office) combining touchscreen pointing, gesturing, touching, and spoken voice recognition commands. Anticipates using wall-sized display, in-air pointing by ultrasonic or capacitive sensor to point to icons. Refers to in-air pointing (hover, proximity) as Teletouch, "fingerprint" confirming cursor to indicate icon is being pointed to. Cites to Micropad handwriting recognition terminal, and Bolt "Put-that-there" gesture input. [Srihari82] Srihari, Sargur N. and Bozinovic, Radmilo "A String Correction Algorithm for Cursive Script Recognition", Proceedings of the 6th International Conference on Pattern Recognition, IEEE CH1801-0/82/0000/ , pp Not sure about conference name Spelling dictionary string correction, using a posteriori computation after recognition is done [Srihari83a] Srihari, S.N., Hull, J.J., and Choudhari, R. "Integrating diverse knowledge sources in text recognition", A.C.M. Trans. Office Information Systems, Vol 1 no 1, 1983, pp Context from bottom-up (probability from previous letter sequence), channel (probability of "A" corrupting to "B"), and top-down (lexicon). Spelling correction for substitution errors: same as Hull83a [Srihari83b] Srihari, Sargur N. and Bozinovic, Radmilo "Use of Knowledge in the Visual Interpretation of Cursive Script", International Conference On Systems, Man and Cybernetics, pages IEEE Computer Society Press, 1983 Cited by Marlin Eller, Microsoft Pen Computing group [Stentiford83] Stentiford, Frederick W.M. and Mortimer, R.G. "Some New Heuristics for Thinning Binary Handprinted Characters for OCR", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-13 No 1, January/February 1983, pp Line-thinning for handwritten data Skeletonization/thinning: side effects are spurious projection, necking, tail generation, noise holes, with heuristics to fix them. [StewartTJ81] Stewart, Theodor J. "An Interactive Approach to Multiple Criteria Decisionmaking Based on Statistical Inference", IEEE Trans. on Systems, Man, and Cybernetics, Vol SMC-11, No 11, November 1981, pp 733-ff feature selection critical in pattern recognition (see other references on choice of features) [Stonham82] Stonham "Networks of Memory Elements -- A Processor for Industrial Automation", Digital Systems for Industrial Automation, Issue No 1 pp 2-3, 1982 Cited in Faggin89: for pattern recognition on human faces Cited in Faggin89: in connection with Wizard/Wisard adaptive image classifier [Suen82] Suen, C.Y. and Shinghal, R. "A Method for Selecting Constrained Hand-Printed Character Shapes for Machine Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol Pami-4 No 1, January 1982, pp Describes Suen's data base of 174 types of sample characters [Suen83] Suen, C.Y. "Handwriting Generation, Perception, and Recognition", Acta Psychologica, Vol 54, 1983, pp Describes printing, script, and manuscript styles and their legibility to humans. Says writing speed for printing increases with practice. Digitizer spec is for 3/8" proximity sensing range. [Summagraphics81a] Summagraphics "Bit Pad One User's Manual", Summagraphics Corporation, August 1981 User's manual for Summagraphics Bit Pad One magnetostrictive tablet digitizer. Describes (accurately?) magnetostrictive technology. Has Schematics of control electronics. [Summagraphics82a] Summagraphics "Summagrid User's Guide", page A-3, Summagraphics Corporation, 1982, Fairfield, Connecticut, 1982 [Summagraphics82b] Summagraphics "An Application Note: How the Summagraphics data tablet digitizer is used", Summagraphics, 1982 Application note for Summagraphics data tablet, pattern recognition for handwritten characters at University of New Brunswick Computing Centre, handwritten characters: pre-processing and classification subroutines, with menu-picking areas (rectangles) on side of tablet. [Summagraphics82c] Summagraphics "Bit Pad One: The versatile Data Tablet Digitizer for your computer system Product Information", Summagraphics, 1982 BitPad-1: de facto serial interface standard for digitizing tablets, especially with stylus. Sales literature 8:26:20 PM]

95 ( ) History of Pen and Gesture Computing mentions data entry, cursor control (mouse/locator input), elimination of typing. [Summagraphics83a] Summagraphics " MM 1201 and MM 961 Technical Reference (Preliminary for Preproduction Product)", Summagraphics Corporation, 1983 Electromagnetic digitizer: describes increment mode, with minimum distance to record movement in stream mode. Compare with press-and-hold? [TanakaHa82] Tanaka, Hatsukazu; Hirakawa, Yutaka; and Kaneku, Seiko "Recognition of Distorted Patterns Using the Viterbi Algorithm", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-4 No 1, January 1982, pp Viterbi algorithm plus a trellis of partial patterns, to do recognition of whole from context of parts [TanakaHi82] Tanaka, Hideo; Uejima, Satoru; and Asai, Kiyoji "Linear Regression Analysis with Fuzzy Model", IEEE Trans. Systems, Man, and Cybernetics, Vol SMC-12 No 6, November 1982, pp Fuzzy sets applied to pattern recognition where classifications overlap. [Tanner83] Tanner, Peter, P. and Buxton, William A.S. "Some Issues in Future User Interface Management System (UIMS) Development", Technical Report, University of Ontario, also in "User Center System Design: New Perspectives on Human-Computer Interaction", Norman, Donald A. and Draper, Stephen W (Ed.), 1986, Lawrence Erlbaum Associates, New Jersey / London Joystick, track-ball, digitizer tablet, with two-handed input [TappertCC82] Tappert, C.C. "Cursive Script Recognition by Elastic Matching", IBM Journal of Research and Development, Vol 26 No 6, November 1982, pp Handwriting cursive recognition by point-by-point distance measurements against "ideal" cursive writing. Does not do well with loops vs cusps, which are forms that transition into each other in handwriting motion and variability. Refers to hook effects due to weak tablets as a big problem: compare with Ward patent and article on digitizer tables. Combines letters in connected script to match whole words (context/dictionary.) [Taylor83] Taylor, I. and Taylor, M.M. "The Psychology of Reading", New York: Academic Press, 1983, pp Cited in Bozinvic89 Bozinovic89 cites for human recognition by outline of word (as shown in WrightG52) [Telesis81a] Telesis Systems Corporation "Telesis Systems Corporation, Business plan", Telesis System Corporation, Concord MA, January 1981 (hardcopy book) Business plan for CAD/CAM company. Product used two displays, one for displaying menus and using lightpen/touchscreen input, the other for the CAD display. [ThomasJJ83] Thomas, James J. and Hamlin, Griffith "Workshop summary: Graphical Input Interaction Technique", printed in Computer Graphics, Vol 5, January 1983, pp Workshop report on graphical input interaction user interfaces: including handwriting recognition GUI [ThornburgDD82a] Thornburg, David D.; Flegal, Robert M.; and Lam, Tat C. "Graphics Pen for Soft Displays", United States Patent 4,318,096, March 2, 1982 Tablet pen with cylindrical off-axis force/pressure transducer in the tip, to measure side-force on the stylus, for graphic arts rendering. (Inventor of KoalaPad) [ToddR81a] Todd, Robert "Apparatus and Method for Recognized a Pattern", United States Patent 4,259,661, assigned to Burroughs Corporation, March 31, 1981 Cited in Bokser88 OCR hardware: recognition via template comparison [Toida83] Toida, T. et al "A Method for Correcting Characters in On-Line Handwritten Input", Showa 58-nendo Denshi Tsushin Sogo Zenkoku Taikai, April 1983 Cited in Sklarew99 patent [TorokGP82a] Torok, G.P. and White, A.B. "Remote Chalk-board automatic Cursor", United States Patent 4,317,956, March 2, 1982 Remote chalkboard / whiteboard / telautograph, display remote cursor showing where the stylus/marker is in proximity (?) so that things can be "pointed to" remotely without marking. Digitizer tablet patent on automatic cursor, mark erasure (recognition strokes removed in electronic ink user-interface), GUI display of eraser in remote whiteboard system. Uses "telautograph" system as term for whiteboard system, but unidirectional whiteboard, only communicating one way. Use of a display/proximity cursor on whiteboard to help other user notice where things are being written, and to "point to" images. [Tsichritzis82] Tsichritzis, D. "Form Management", CACM, Vol 25 No 7, July 1982, pp Quoted in Hakmatpour86 (missing from this list?) Deals with forms in office management, says flow of forms is important Mentions non-paper "forms" for voice Refers to operations on forms: automatic calculation, etc. [TsikosC82a] Tsikos, Constantine "Capacitive Fingerprint Sensor", United States Patent 4,353,067, October 5, 1982 Capacitive fingerprint sensor: array of individual capacitors, finger ridges push plates closer together, alternatively sensing plates capacitance to flexible conductive sheet deformed by finger ridges. Later cited 8:26:20 PM]

96 ( ) History of Pen and Gesture Computing in touchpad design. [Tsuruoka83] Tsuruoka, S., Kimura, F., Yoshimura, M., Yokoi, S., and Miyake, Y. "Thinning Algorithms for Digital Pictures and Their Application to Handprinted Characters Recognition", Trans. of IECE of Japan, Vol J66D No 5, May 1973, pp (in Japanese: abstract only) Skeletonization / line-thinning [Turba81] Turba, T.N. "Checking for spelling and typographical errors in computer-based text", in Proceedings of A.C.M. SIGPLAN SIOGA Symposium on Text Manipulation", Portland, Oregon, June 1981, pp Cited in Hull83a spelling/context correction using structural information: invalid prefix ("in" for "im", joining of suffix (beginner vs beginner), etc. rather than just comparison with word list. [Voiers83] Voiers, W.D. "Evaluating Processed Speech using the Diagnostic Rhyme Test", Speech Technology, January/February 1983, pp Speech: how contextual factors are controlled in test protocols affects recognition results. Speech recognition failures completely explained by a limited set of underlying phonemic (human recognition) features [Wakahara83] Wakahara, T. and Umeda, M. "Stroke-number and Stroke-order Free On-line Character Recognition by Selective Stroke Linkage Method", Proceedings of ICTP '83, Tokyo, October 17-19, 1983, pp Combinatorial solution to stroke connection and stroke order variations. Break strokes into equal-length pieces, then match them as a feature [WalkerJ83] Walker, John "The Autodesk File: Crisis Letter", From collection of internal documents on the history of AutoCad: June 21, cites special concern on marketing deal with Sun-Flex (touch-pen touchscreen digitizers), Touch-pen considering alternative vendor P-CAD. [WallaceBA81a] Wallace, Bruce A. "Merging and Transformation of Raster Images for Cartoon Animation", Computer Graphics, vol 15 No 3, August 1981, pp opacity mask and image blending (similar to alpha blending) for anti-aliasing [Wang83] Wang, C. Sun, H., Yada, S., and Rosenfeld, A. "Some experiments in relaxation image matching using corner features", Pattern Recognition, Vol 16, 1983, pp Cited in Yu90: on context for line thinning using chain codes [Watanabe83] Watanabe, Y., Gyoba, J., and Maruyama, K. "Reaction time and eye movements in the recognition task of hand-written Katakana-letters", Japanese Journal of Psychology, Vol 54 No 1, pp 58-61, 1983 (in Japanese) Uses eye fixation to determine what features are cognitively important Repeats and continues Blesser et al's early work from Massachusetts Institute of Technology [Watari83] Watari, Masao, Sakoe, Hiroaki, Chiba, Seibi, Ishizuka, Hisao, Kawakami, Yuichi, and Iwate, Toshiki "A DP-Matching LSI for Speech Recognition", NEC Research and Development, No 70, pp 71-78, July 1983 Dedicated dual processor chip used for speech recognition [WeltyC81a] Welty, Charles and Stemple, David W. "Human Factor Comparison of a Procedural and a Nonprocedural Query Language", ACM Trans. Database Sys. 6(4): (Dec 1981) TABLET - Query language, more procedural than SQL. Human Factors (learnability) of a programming language. TABLET is more verbose, Cobol-like syntax. [WhiteJM83] White, J.M., and Rohrer, G.D. "Image Thresholding for Optical Character Recognition and Other Applications Requiring Character Image Extraction", IBM Journal of Research and Development, Vol 27 No 4, pp , July (Abstract only) Handwriting recognition using nonlinear adaptive procedure: thresholding of scanned OCR images Preprocessing to clean up OCR scanner images of carbon copy forms, bank checks, smudges, scenic backgrounds, etc. [WhitfieldD83] Whitfield, D., Ball, R.G. and Bird, J.M. "Some comparisons of on-display and off-display touch input devices for interaction with computer generated displays", Ergonomics, Vol 26, 1983, pp Beringer89, electronic ink hardware? Includes definition of touchscreen and touchpad which may be separate from display, review of touch-screen digitizer technologies. Mentions position accuracy errors at light touch or at end of a stroke as finger is lifted. Touchpad uses mylar sheets held apart by air pressure: micropad? Fall-out errors (inadvertent skips) on touchpad/tablet [WillisR83a] Willis, Richard "Big Blue Goes Japanese", BYTE Magazine, Novmeber 1983, pp IBM 5550 Japanese (Katakana and Kanji) display terminal, 24x24 matrix characters. See also CIC Handwriter handwriting recognition tablet for [Wing83] Wing, A.M., Nimmo-Smith, M.I., and Eldridge, M.A. "The Consistency of Cursive Letter Formation as a Function of Position in the Word", Acta Psychologica, Vol 54, 1983, pp Allograph (variant) selection based on preceding context character [Witkin83] Witkin, A. P. "Scale-space filtering", Proceedings of the International Joint Conference on Artificial Intelligence, 1983, pp :26:20 PM]

97 ( ) History of Pen and Gesture Computing Cited in Lipscomb91 Template matching of varying signals by filtering/smoothing at different scales to reduce noise, filtering is adaptive and variable [WolfeldJA81a] Wolfeld, Jeffrey A. "Real time control of a robot tactile sensor", Technical report MS-CIS and Master's Thesis, Dept. of Computer and Information Science, U. Penn., August 1981 Early multi-touch sensor (cited in Lindemann): Pad sensor, 8x8 grid of pressure (force) sensitive sites, connected to strain gauges for parallel force. References for artificial force-sensing/touch-sensing robotic skin. [WuL82a] Wu, Li-De "On the Chain Code of a Line", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-4, No 3, May 1982 Chain codes of optically scanned / static character recognition OCR [Xerox81a] Xerox Corporation "Xerox 8010 Star Information System Learner's Guide", Xerox Corporation, 1981 Introductory User's Guide to Xerox Star. printing, mailing, document editing, ford processing, etc. Pointand-click with mouse, no drag gestures. [YamamotoE81] Yamamoto, E., Fujii, N., Fujita, T., Ito, C., and Tanahashi, J. "Handwritten Kanji Character Recognition Using the Features Extracted from Multiple Standpoints",?? Conference Proceedings, IEEE CH /81/0000/0131, 1981 Using multiple features rather than single feature, and weighting results, gives better performance on character recognition. [YamamotoK83] Yamamoto, K. "Studies on the Recognition of Handprinted Characters by Structural Analysis Methods", Research Electrotechnical Laboratory, Report No , February 1973 (in Japanese) Outermost point recognition method for Romanji, numeral, Katakana [Yamasaki82] Yamasake, T., Inokuchi, S., and Sakurai, Y. "Training System for Handwritten Chinese Characters Using On-Line Character Recognition Techniques", Trans. of IECE of Japan, Vol E65 No 10, p 602, October 1982 (Abstract only) Science Citation Index Well-writing: teaching users how to write Chinese, not recognize Chinese handwriting [YasuharaM82a] Yasuhara, M. and Yasumoto, Y. "An Improved Adaptive Predictor in DPCM Based on Kalman Filter and Its Application to Handwriting Signal Encoding", Working paper submitted to IEEE Trans. on Comp., Univ. of Electro-Commuications, Japan, March 25, 1982 Encoding of handwriting motion/motor input based on model of handwriting motion [YasuharaM83a] Yasuhara, Makoto "Identification and Decomposition of Fast Handwriting System", Working paper, to appear in IEE Trans. on CAS, Vol 30 No 11, Novm 1983 Model of handwriting dynamics motion for fast handwriting (more dynamic) [Yedwab81] Yadweb, Laura; Herot, Christopher F.; Rosenberg, Ronni L.; and Gross, Carol "The Automated Desk", SigSmall Newsletter, Vol 7 No 2, October 1981, pp Shneiderman83 cites for direct-manipulation desktop user interface. Desktop UI displayed on charactermode terminal. No stylus/digitizer input shown, uses joystick, arrow keys, keyboard. [Yhap81] Yhap, E.F. and Greanias, E.C. "An On-Line Chinese Character Recognition System", IBM Journal of Research and Development, Vol 25 No 3, May 1981, pp National differences/variability in writing styles (Japanese, Chinese, Korean) Claimed 97.8% accuracy by excluding 5% of data as "poorly written" Stroke order, stroke connection variations in Chinese The 214 standard Chinese radicals "too many": some are rare Recognition diagram: signal filter, segment and direction, stroke Classification, alphabet/element recognition, composite ideograph output Contrast: features of primitive "stroke element" recognition vs chain code segments [Yoshida82] Yoshida, K., and Sakoe, H. "Online Handwritten Character Recognition for a Personal Computer System", IEEE Trans. Consumer Electronics, Vol CE-28 No 3, pp , August 1982 System connects strokes of "normal" to make "running" forms (variability?), feature is angle sequence (chain codes?) Claims 99.5% accuracy on handwriting recognition [Yoshida83] Yoshida, K., and Sakoe, H. "Online Character Recognition by Stack DP Matching Method", PRL83-29, Sep. 27, 1983 (in Japanese, abstract in English) Character represented as branches in a reference pattern feature sequence. Detail "discrimination logics" for character which the regular method cannot handle. Kanji and Hiragana handwriting biblio80 (Prev page) Main Page (Next page) biblio85 In real life, there are no silly questions: there are only silly questioners. Wenn Geld nicht glücklich macht, warum freuen sich die Leute so wenn sie beim Lotto gewinnen? 8:26:20 PM]

98 ( ) History of Pen and Gesture Computing 8:26:20 PM]

99 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1984 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:28 EDT Mirrors of this page can be found at: (Generally most up-to-date) biblio83 (Prev page) Main Page (Next page) biblio86 [ANSI84] ANSI "Draft Proposed American National Standard Graphical Kernel System", American National Standards Institute, Incorporated, X x, 1984 GKS graphics API standard [Ahmed84] Ahmed, P. and Suen, C.Y. "Edge classification and extraction of shape features", Proceedings of the 7th International Conference on Pattern Recognition, 1984, pp Cited in Li89 [Aleksander84] Aleksander, I., Stohnam, T.J., and Wilkie, B.A. "Recognition Apparatus", United States Patent 4,490,847, December 25, 1984, assigned to National Research Development Corporation, London, England. Video image recognition, based on learning (neural) networks [AllesHG85a] Alles, Harold G. and Kubik, Peter S. "Deformable Touch Sensitive Surface", United States Patent 4,542,375, September 17, 1985 Touchscreen digitizer using transparent deformable surface (transparent rubber) over a CRT display. Principle somewhat similar to light pen: where surface is pressed, it bends closer or makes contact with the front of the CRT display, causing light from the electron beam to be reflected by total internal reflection. Photodiode light sensors on the side detect the light, and from the (timed) position of the CRT spot (similar to light pen), know the location. Should be able to detect multiple touches, since reflected light is detected all around the display. [Apple84] Johnson, Lynnea "Macintosh MacWrite", Apple Computer Inc, 1984, Product Number #M1502 (hardcopy book) User manual and introduction to MacWrite word processing application [ArcelliC85a] Arcelli, C. and Sanniti di Baja, G. "A Width-Independent Fast Thinning Algorithm", IEEE Trans. Pattern Analysis and Machine Intelligence", Vol PAMI-7 No 4, July 1985, pp Skeletonization/line-thinning with a method to "recover" the original outline by building the skeleton back up Line thinning by topology-preserving removal operations [ArtwickB84a] Artwick, Bruce "Applied Concepts in Microcomputer Graphics", Prentice-Hall, 1984, ISBN :26:23 PM]

100 ( ) History of Pen and Gesture Computing Describes viewports and Windows as projected coordinate calibration on a digitizing tablet. Early CAD menus printed statically on the digitizer surface as a template. [Asija85] Asija, Satya P. "Automated Image Input, Storage and Output System", United States Patent 4,523,331, June 11, 1985 Automated image input, including handwriting recognition, and output, including generation Shows characters normalized to a 5x7 grid, and stored based on bit string of traversal of squares Recognition trained to each user: matches are by binary-value distance between two bitstrings Note: Compare with input technology of Casio PF-8000 calculator: Casio84a_PF8000 [Autodesk84] Autodesk "AutoCad 2 (tm) Drafting Package User Guide", # , Autodesk Incorporated, 2320 Marinship Way, Sausalito CA 94965, 1984 [BaeckerR85a] Baecker, Ronald Michael "Curriculum Vitae, Ronald Michael Baecker", Ronald Maechel Baecker, Nov 29, 1985 CV and publication list as of 1985 for Ronald Baecker. Founder of HCR, Unix-based software start-up. [Baird84] Baird, H.S. "Model-based Image Matching Using Location", PhD Thesis, Department of Electrical Engineering and Computer Science, Princeton University, October 1984 Pattern matching by a template in feature-space, looking for minimum distances of feature points in a multi-dimensional vector. Mathematical pattern-matching thesis, without any examples of any specific patterns it was tried on! [BarkerPG84] Barker, P.G., Najah, M. and Roper, J.S. "User Experiences with a MICROPAD", Journal of Microcomputer Applications, Vol 7, 1984, pp Cited in BarkerPG85 [BarkerPG85a] Barker, P.G. and Janah, M. "Implementing pictorial interfaces using a high-resolution digitizer", International Journal of Man-Machine Studies, Vol 23, 1985, pp User interface picking menus from parts of pictures (graphical menus) -- happened to use a Micropad digitizer, not really handwriting recognition [BarkerPG85b] Barker, P.G. and Janah, M. "Pictorial interfaces to data bases", International Journal of Man- Machine Studies, Vol 23, 1985, pp User -interface using menus on picture parts, not lists, pick-boxes from a Micropad terminal [BeckerJ84] Becker, Joseph D. "Multilingual Word Processing", Scientific American, Vol 251 No 1, July 1984, pp Character/word order, user-interface design for multiple language word processing (Arabic/Hebrew/English/Roman/Chinese) Korean is written in syllable clusters, Hebrew right-to-left, Arabic right-to-left with words diagonally, Mongolian vertically [BeckerJ85] Becker, Joseph D. "Typing Chinese, Japanese, and Korean", IEEE Computer, January 1985 Keyboard input for Chinese Kanji [Belaid84] Belaid, Abdelwaheb and Haton, Jean-Paul "A Syntactic Approach for Handwritten Mathematical Formula Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-6 No 1, January 1984, pp Refers to ambiguity of nominally straight vs curved lines Mentions P/D, T/+ ambiguity Investigate highlevel context and ambiguity in mathematical formulas Refers to min/max extents, centroid of characters Says some irresolvable "confusion" ambiguity solved by user interaction User interface for 2-D mathematical symbol input: handwritten sketch input [Berkin84] Berkin, G.M., and Negin, Michael "Pattern Recognition System", United States Patent 4,441,205, April 3, 1984, assigned to Kulicke and Soffa Industries, Incorporated, Horsham, Pennsylvania. Alignment pattern recognition for video scanning of integrated circuits, small parts [BinaryFusion85a] Binary Fusion Software Archive "Borland Sidekick", Fetched October 2010 Downloadable install files (zip archive) for Borland Sidekick: files dated Mentions feature of onscreen dialer for telephone numbers.. [BirrellAD84a] Birrill, Andrew D. and Nelson, Bruce Jay "Implementing Remote Procedure Calls", ACM Trans. on Computer Systems, Vol 2 No 1, February 1984, pp Design review of RPC implementation on Cedar: cites to RFC Practical problems of network failure, address/pointer parameters (to be marshalled), binding to target/callee, authentication and security: PKI/encryption for authentication etc. [Blesser84] Blesser, Barry "Comments on Variability due to Pen-Artifacts", internal report, Pencept, Incorporated September, 1984 Apparent variability of handwriting in on-line recognition caused in part by defects/performance of tablet digitizers: retrace, hooks, etc. [BlesserB85a] Blesser, Barry "Tablet Error Analysis", internal report, Pencept, Incorporated September, 1985 Mathematical measurements of tablet performance, focusing on defects and quirks of different types of digitizing tablets: describes effects of search algorithms used to locate stylus, local stretch, external 8:26:23 PM]

101 ( ) History of Pen and Gesture Computing voltages, electronic defects, etc. [BlesserB85b] Blesser, Barry "Tablet Technologies", internal report, Pencept, Incorporated September, 1985 Review of tablet technologies: hundreds of types, multiple categories -- resistive=spatially continuous; grid electrostatic/electromagnetic=spacially discrete, pulse excitation, sine-wave continuous excitation, contact control (tip switch): cases study of Pencept tablet technology. [BoieRA84a] Boie, R. A. "Capacitive Impedance Readout Tactile Image Sensor", Proc. Int. Conf Robotics. Washington, D.C., 2004, pp (abstract only) 8x8 / 32x32 capacitive tactile sensor touchpad for robotics, matrix of X and Y conductors measuring capacitance change at each crossing to form an image. Cited by Buxton for early multi-touch. [BoieRA85a] Boie, Robert A. and Miller, Gabriel L. "Conformable Tactile Sensor", United States Patent 4,526,043, July 2, 1985 capacitive tactile sensor touchpad for robotics or touchscreen; center of areas of contact easily determined; elastomeric/elastic dielectric material between two layers of grid, matrix of X and Y conductors measuring capacitance change at each crossing to form an image. Multi-touch. Multiplexing is analog. [BoiesSJ85a] Boies, Stephen J., Gould, John D., Levy, Stephen, Richards, John T., Schoonard, Jim "The 1984 Olympic Message System -- a Case Study in System Design", IBM Research Division Report RC (#50065), 5/2/85, 53 pages Design study of early voic /messaging system prototype for 1984 olympics. Emphasizes empirical testing and design iteration for usability -- compare with Agile methodology? [BoivieRH84a] Boivie, Richard H. "Directory look-up method and apparatus", United States Patent 4,453,217, June 5, 1984 Spelling correction by recursive division of the dictionary: skip one character at a time: interesting Dictionary searching method [BoivieRH85a] Boivie, Richard H. "Character Recognition Arrangement", United States Patent 4,525,860, June 25, 1985 Described curve tracer/follower for OCR of handwritten characters OCR using line thinning, then simulated chain codes generated by a curve tracer/follower: claims to be font-independent / handwriting recognition "standard spelling": coded sequence of chain codes Refers to two approaches for character recognition: theoretic (or discriminant) versus syntactic (or structural) pattern recognition approach: dislikes both Subpatterns (features) of a handwritten character can only be recognized in the (graphical) context of the entire character pattern Thinning handwritten characters and comparing with dictionary of chain codes for recognition? [BonoPR85a] Bono, Peter R. "A Survey of Graphics Standards and Their Role in Information Interchange", IEEE Computer, Vol 18 No 10, October 1985, pp User-interface management system (UIMS) and graphics standards: GKS, PHIGS, etc. See also Bono presentation on handwriting character recognition in standards. [BonoPR85b] Bono, Peter R. "Software Standards: Which Ones Are Here to Stay?", S. Klein Computer Graphics Review, Inaugural Issue, 1985 Permanence/effect of standards depends on whether major manufacturers support them, base produces on them: examples GKS Graphics Kernel System and CGM Computer Graphics Metafile. Overview of standards-making process (GSK, PHIGS, etc.) [Borland84a] Borland International "Sidekick Version 1 Owner's Handbook", Borland International, 1984 PIM personal information manager for DOS: calculator, notepad, auto-dialer (finds telephone numbers by screen scraping). Character-mode windows. [Borland85a] Borland International "Sidekick Version 1.5 Owner's Handbook: The Desktop Organizer just a Keystroke Away", Borland International, March, 1985 PIM personal information manager for DOS: calculator, notepad, auto-dialer (finds telephone numbers by screen scraping, or from telephone directory file / contact database) using Hayes 1200/1200B modem. Character-mode windows, windows auto-rearrange. [BozinovicR84a] Bozinovic, R. and Srihari, S.N. "Knowledge-based Cursive Script Interpretation", Proceedings of the 7th International Conference on Pattern Recognition, Montreal, 1984, pp Cites own poor performance if did not exclude "hill and dale" sloppy writing, crowded writing Assumes a premise that people use script, not printing, in normal writing [BraultJJ84a] Brault, J.J. and Plamondon, R. "Histogram Classifier for Characterization of Handwritten Signature Dynamic", Proceedings of the 7th International Conference on Pattern Recognition, Montreal, 1984, pp Signature verification: small sample size: 243 signatures of 50 people: test protocol had no forgery attempts (!) [BritishMicro85a] Micro-Draw "GRAFPAD 2 Art for the AMSTRAD by Micro-Draw", Micro-Draw, UK, 1985 Drawing program for GRAFPAD low-cost / low-resolution digitizer tablet. On-table printed overlays for commands -- compare with PenCAD, CIC CAD. No gestures. 8:26:23 PM]

102 ( ) History of Pen and Gesture Computing [BrocklehurstE85a] Brocklehurst, E. "Computer methods of signature verification", Journal of Forensic Science, Vol 30, 1985, pp Cited in Yoshimura91 [BrogardhT84a] Brogardh, Torgny "Device for Simultaneous Creation and Detection of Markings", United States Patent 4,475,240, October 2, 1984, assigned to ASEA Aktiebolag, Vasteras, Sweden. Digitizer tablet using a light-sensitive optical sensor to detect markings as they are written/drawn [BrogardhT85a] Brogardh, Torgny "Optical Digitizer/Position Measuring Device", United States Patent 4,531,230, July 23, 1985, assigned to ASEA Aktiebolag, Vasteras, Sweden. Fiber-optic optical digitizer tablet using a grid of an optically-sensible pattern [Brooks85] Brooks, C.P. and Newell, A.F. "Computer Transcription of Handwritten Shorthand as an Aid for the Deaf -- a Feasibility Study", International Journal of Man-Machine Studies, Vol 23, 1985, pp Character recognition for shorthand DCR for shorthand much more complex than was thought 1/4 to 1/2 of words in spoken conversation unintelligible out of context Hooks, loops, circles are "real" features in shorthand Refers to endpoint trimming on digitizer strokes "invariant" base form, and numerous real written forms for one writer Hard to write vertical curved strokes reliably [BrownG85] Brown, G.P., et al "Program Visualization: Graphical Support for Software Development", IEEE Computer, Vol 18 No 8, August 1985, pp Interactive graphical input and display interface languages for programming: compare with layout editors? Digitizer using separately mounted Lexan sheet over physical display (with shift in position?) [BrownMR85a] Brown, Mark R.; Kolling, Karen N.; and Taft, Edward A. "The Alpine File System", ACM Trans. Comp.Systems, Vol3 No 4, November 1985, pp file system similar to Multics file system: part of Cedar system. Access control list uses sequence syntax of RNames for users and/or groups with type of access granted (limit to 20 RNames in syntax): RName groups can be nested hierarchy expression. [BrownRM85] Brown, Robert M. "Handprinted symbol recognition system", United States Patent 4,491,960, January 1, 1985, assigned to the United States of America as represented by the Secretary of the Navy. Cited in Sklarew88b: on OCR of handwriting? uses logic tree of robust features first, then lesser discriminants, then a quality-assurance module filtering/noise-removal: (1) pre-recognition ("trash" filter), (2) potential symbol identification, and (3) final quality assurance; for on-line handwriting recognition [BuxtonW85] Buxton, William "Notes on Chord Keyboards", Computer Systems Research Institute, Univ. of Toronto, Toronto Ontario CA M5S 1A4 mentions WriteHander chord keyboard from NewO Corporation [BuxtonW85a] Buxton, William, Hill, Ralph, and Rowley, Peter "Issues and Techniques in Touch-Sensitive Tablet Input", SIGGRAPH 85, San Francisco July 22-26, Vol 19 No 3, 1985, ACM /85/007/0215, also Computer Graphics Vol 19 No 3. Also on billbuxton.com: fetched 2000 Describes virtual tablets, multiple virtual devices on a tablet. Use of pressure/force and multiple-touch input devices. Refers to inaccuracy ("jitter") of touch tablets / touchscreens at light pressure, such as at end of stroke/touch-lift, but does not mention inaccuracy of lifting motion. (Minimum distance for pressand-hold). Virtual switch device, compare with Apple slide/swipe-to-unlock gesture. Advantages of multiple virtual devices on a display, for which multi-touch input would be especially advantageous. Two levels of pressure/force (haptic?). Painting with pressure-sensitive tablet. Color mixing with multiple touch sliders. Strong advantages to using multi-touch input. Also: Electronic video file of demonstration. Cited in Wanderley00, but with wrong authors? [BuxtonW85b] Buxton, W., Sniderman, R., Reeves, W., Patel, S. and Baecker, R. "The evolution of the SSSP score-editing tools", In Foundations of Computer Music, C. Roads and J. Strawn, Eds, MIT Press, Cambridge, Mass 1985, ch 22, pp ; Originally published in Computer Music Journal, Vol 3 No 4 Issue 12, December 1979; pp 14 ff. Gestures/marks for editing musical scores, handwriting music recognition. Recognition based on changes in angles only, not lengths of segments. Semantically uses multiple segments to indicate note duration, but does not treat as compound recognition. Shapes later copied by PLI group at IBM for music editing demo. Also available at [BuxtonWB85d] Buxton, William "A Multi-Point Three-Dimensional Touch Sensitive Tablet", University of Toronto, 1985: available at billbuxton.com Video to accompany "A Multi-Touch Three Dimensional Touch-Sensitive Tablet". Lee's capacitive-matrix multi-touch tablet. [Byte84a] BYTE Magazine "Real World Interfacing", BYTE Magazine, Vol 9 No 4, April 1984 Pencept advertisement for handwriting recognition tablet for IBM PC: "Is the Penpad 320 the greatest graphics tablet in the world? Or, is it the greatest keyboard and mouse? [Byte85a] BYTE Magazine "Bargain Computing", BYTE Magazine, Vol 10 No 3, March 1985 Space Tablet three-dimensional/3d sonic digitizer pointer, Soniture Inc. KoalaPad digitizer tablet. 8:26:23 PM]

103 ( ) History of Pen and Gesture Computing [CIC84a] CIC "Handwriter (TM) Product Literature (in Japanese)", CIC Japan Incorporated, Suite 918, Shuwa Kioicho TBR Building, 5-7 Kojimachi, Chiyoda-Ku, Tokyo 102, Japan, 1984 Forms data entry handwriting recognition product from CIC Handwriter with tablet. States 99% correct recognition, seconds per character input speed. [CIC84b] CIC "The Handwriter: A Quiet and Natural Data Entry System", Communication Intelligence Corporation, Menlo Park, CA, 1984 (date approximate) Information in English on CIC Handwriter recognition tablet. [CIC85a] CIC "Handwriter (R) focus:abc (TM) Accounting Package", Communication Intelligence Corporation, 800 El Camino Real, Menlo Park, CA 94025, 1985 (hardcopy file) Spreadsheet from CIC Handwriter product [CIC85b] CIC "Handwriter (R) Lotus interface Kit", Communication Intelligence Corporation, 800 El Camino Real, Menlo Park, CA 94025, 1985 (hardcopy file) Word-processing from CIC Handwriter product [CIC85c] CIC "Handwriter (R) WordStar (R) interface Kit", Communication Intelligence Corporation, 800 El Camino Real, Menlo Park, CA 94025, 1985 (hardcopy file) The ASCII handwriting recognition product Hew Crane said did not exist [CIC85d] CIC "Handwriter (R) GrafText (TM) System Model GT-5000 / Host Communication Protocol Manual", Communication Intelligence Corporation, 800 El Camino Real, Menlo Park, CA 94025, June 1985 (hardcopy file) Similar product to Pencept PenPad. Describes recognition modes and templates for forms: graphic, text, remote-function, local-function areas. Calibration of tablet to paper form. Handwriting recognition returned multiple matches (up to 5). Electronic ink display in tablet driver. [CItoh85] CItoh Corp "CI-Handwriter Fit-In-1 product information", (OEM version of CIC Handwriter product). Japanese handwriting character recognition product [CPCWiki85a] CPCWiki "Grafpad II Graphic Tablet", Amstrad product wiki, fetched 2015 Low-resolution (1000x1500) digitizer tablet for early IBM PC, etc. Updated version of Grafpad by British Micro. On-tablet overlays for commands, virtual keyboard. File includes some break-down pictures of circuit/sensor board: looks like X and Y grid on opposite side of circuit board. Cites to FutureOS software by Dr. Stefan W. Stumpferl, Micro-Draw manual (see). [CalComp85a] CalComp "CalComp 2500 Series Digitizer Operator's Manual User's Guide", CalComp Incorporated , 1985 [CalComp85b] CalComp "CalComp 2500 Series, 9500 Series Digitizer product information", CalComp Incorporated CalComp digitizer tablet: shown with stylus, four-button cursor/puck, 16-button cursor/puck with reticule [CardelliL85a] Cardelli, Luca and Pike, Rob "Squeak: a Language for Communicating with Mice", Proc. SIGGraph '85, San Francisco, pp Tutorial on Squeak, programming language for UI implementation using multiple input streams e.g. keyboard input stream and mouse event input stream. "Menu" event on press-and-hold, when mouse button depressed for a significant time (presumably without being moved). [CarnesWR84a] Carnes, W.R. "Signature Verification System", United States Patent 4,443,436, February 21, 1984 From patent list [Carpenter85] Carpenter, G.A. and Grossberg, S. "Neural Dynamics of Category Learning and Recognition: Attention, Memory Consolidation, and Amnesia", in "Brain Structure, Learning, and Memory", Davis, J., Newburgh, R., and Wegmen, E. editors, AAAS Symposium Series, 1985 Adaptive recognition: refers to "as recognition categories stabilize" in learning [CarrTH84] Carr, T.H., Brown, T.L., Brown, J.S., and McDonald, J.L. "Handwriting of connected discourse: Speedaccuracy tradeoffs in a skilled performance", Proceedings of Psychonomic Society, November, 1984 Cited in Rhyne86: : was in Tappert's group 1986 Rhyne86 cites this that mental set affects legibility of handwriting (variability) [Casey84a] Casey, R.G. and Nagy, G. "Decision Tree Design Using a Probabilistic Model", IEEE Trans. Information Theory, Vol IT-30 No 1, January 1984, pp Applications: single-font OCR system had to replace five keypunch operators at prices of 1975 Parallelism vs serial: decision tree is best technique for speed/cost tradeoffs in character recognition Paradigm: iterative optimization of decision tree by designer vs automatic construction of decision tree (designer specifies features to use) Decision trees in OCR: for small trees, need very uniform styles and pixel data (is not that obvious?) Estimated versus actual performance: adding more OCR pixels does not help much since they are not significant features (size of character/resolution) Testing: reports real results on trading reject for substitution errors [Casio84a] Casio "Casio PF-8000 Calculator user manual (in Japanese)", Collection from g-mark.org, Calculator with touchscreen input, and also zone-based handwriting recognition on a touch-film 8:26:23 PM]

104 ( ) History of Pen and Gesture Computing keyboard/keypad. Also two physical devices in collection. Not sure about relationship of PF8000 and IF8000 products unistroke / single-stroke characters (?) Dr. Theodore Kuklinski had one in his collection in 1990 Additional material in file: Casio/Casio Data Bank PF-8000: full-frame picture Additional reference on Casio Data Bank PF-8000: showing writing styles [Casio84b] Casio "Casio PF-8000 Calculator user manual (in English and Spanish)", Casio 123B M publication, date believed to be Calculator with touchscreen input, and also zone-based handwriting recognition on a touch-film keyboard/keypad, separate from the display Shows a "gesture" character for editing (e.g. backspace/delete), demonstrated in a name/address-book application included in the unit. Functional unit in personal collection of Jean Renard Ward, this is the user documentation. Unistroke / single-stroke characters. See also video files. Includes a reference sheet for writing single-stroke characters, similar to the "Graffiti" unit by U.S. Robotics in the 1990s [Casio84c] Casio "Operation Manual Casio PF-8000 Calculator (in English)", Casio, Calculator with touchscreen input, and also zone-based handwriting recognition on a touch-film keyboard/keypad. [Casio84d] Casio "Casio Electric Notebook PF-8000", 2.html (fetched 2002) Description (in Japanese) of 1984 Calculator with touchscreen input, and also zone-based handwriting recognition on a touch-film keyboard/keypad [Casio84e] Casio "Casio PF-8000 Calculator", In personal collection of Jean Renard Ward -- cover missing, approximately 34 pages (physical device) 1984 Calculator with touchscreen input, and also zone-based handwriting recognition on a touch-film keyboard/keypad [Casio84f] Casio "Casio TC-500 touch screen calculator watch product information", Casio 1984 wristwatch with touchscreen input, handwriting recognition for digits and arithmetic operators. See also TC-550, TC-600. File contains user guide. [Casio84g] Casio "Casio AT-550 Touch Screen Calculator Watch: Video", Casio Video of 1984 wristwatch with touchscreen input, handwriting recognition for digits and arithmetic operators. Uploaded to YouTube.com by Bill Buxton, [Casio84h] Casio "Casio PF-8000 Data Bank PF-3000 Data Bank Screen-Write your data", Casio Photo brochure of Casio PF-8000 Data Bank touchscreen/handwriting calculator/pda: shows details of touchscreen [CasioCalculatorWatch84a] Casio Calculator Watch "Vintage Casio TC-600 Touch Screen Calculator Retro LCD wristwatch", casiocalculatorwatch.net, fetched 2010 (linked index) Casio wristwatch with unistroke character input on touchscreen, circa 1984: indexed under 2010 (date fetched) [Castleberry85] Casteleberry, Donald E. and Becker, Charles A "Electroscopic information display and entry system with writing stylus", United States Patent 4,520,357 May 28, 1985, assigned to General Electric Company Sklarew cites as electroscopic/electrostatic tablet/display integrated device Capacitive-coupling digitizer (with a magnetic stylus) integrated with a matrix display [Chainer85a] Chainer, T.J., and Worthington, T.K. "Segmentation Algorithm for Signature Verification", United States Patent 4,553,258, November 12, 1985, assigned to International Business Machines, Armonk, New York. Finds pen lift/down stroke start/end via A.C. coupling to a force/pressure gauge [Chainer85b] Chainer, T.J., Gundersen, S.C., and Worthington, T.K. "Semi-Independent Shifting Technique for Signature Verification", United States Patent 4,553,259, November 12, 1985, assigned to International Business Machines, Armonk, New York. Force/pressure and acceleration information on signature verification correlating data by shifting it [Chainer85c] Chainer, T.J., Scranton, R.A. and Worthington, T.K. "Data Input Pen for Signature Verification", United States Patent 4,513,437, April 23, 1985, assigned to International Business Machines, Armonk, New York. Digitizer stylus pen with force/pressure and acceleration sensitivity for signature verification: Better force/pressure-sensitive (piezo-electric sensor) pen stylus than Herbst 4,142,175 [Chainer85d] Chainer, T.J. and Worthington, T.K. "Shifting Technique for Signature Verification", United States Patent 4,562,592, December 31, 1985, assigned to International Business Machines, Armonk, New York. Shifting (to reduce correlation computation) of segments of signatures for acceleration and force/pressure data for signature verification [Chalkboard84a] Chalkboard "Electronic Chalkboards: Powerpad Product Information", Chalkboard Inc, 1984 Educational software using low-resolution digitizer tablet for VIC-20, Commodore 64, IBM-PC (DOS), Apple-II and Atari computers. User interface is mylar overlays with pre-printed simplified keys for child use. Educational software include Mico Maestro for music, 'Lectric Paintbrush, Music Math. Additional material in 8:26:23 PM]

105 ( ) History of Pen and Gesture Computing paper file [ChenCG85a] Chen, Chee G. "Transparent Touch Switching System", United States Patetn 4,516,112, May 7, 1985 Piezo-electric touchscreen, piezoelectric polymer film pressed towards screen, transparent flexible conductive coatings on both sides of film as electrodes. Transparent flexible non-conductive/insulating film as cover. [Cheung85] Cheung, Y.S. and Leung, C.H. "Chain-code transform for Chinese character recognition", Proceedings of IEEE International Conference on Systems, Man, and Cybernetics", Tucson, Arizona, 1985, pp Japanese/Kanji/Katakana product from Hew Crane and James Dao at CIC: Cited in Leung87 [ChunK85a] Chun, Kukjin and Wise, Kensall D. "A High-Performance Silicon Tactile Imager Based on a Capacitive Cell", IEEE Trans. on Electron Devices, Vol ED-32 No 7, July 1985, pp / Technical report 187, U. Michigan, May 1986 Tactile sensor for robotic touch: force/pressure capacitor of deformable pad/outer skin over silicon capacitive sensors in 8x8 array, may be attached to glass, may be combined to give 16x16 array. Capacitive matrix touchpad sensor, intended for robotics applications. Matrix of capacitive force sensors on 2mm centers, produces X/Y image of touch pressure (hence the name). Compare with Rekimoto multitouch tablet. [Cobblestone85] Cobblestone Design "Pencept: Software Control at the Stroke of your Pen / Write into your computer", Photograph of trade-show booth from Comdex for Pencept, handwriting recognition. [ComerfordR84a] Comerford, Richard "Pointing-device innovations enhance user/machine interface", Electronics Design News, July 26, 1984, pp (hardcopy book) Appears in Sherr 1988 "Input Devices". Review of digitizer/tablet/joystick/mouse/touchscreen/trackball vendors, 1984 [Commodore84a] Beekman, George "The Commodore 64 Home Companion", DataMost, 1984, illustrations by Martin Cannon Instruction and software catalog for Commodore 64 home computer, and various third-party software and devices. Describes KoalaPad touch digitizer, and also ChalkBoard Power Pad: Power Pad uses different construction and can sense and report more than one touch at a time: multi-touch. [ComputerMuseum84a] The Computer Musecum "The Computer Museum Report Winter: Grand Re-opening", The Computer Museum Report, Winter 84/85 Vol 11 TCMR-V11 Report on Computer Museum physical exhibits: includes original RAND tablet digitizer, GRAIL (GRAphical Input Language) programming, machine recognition of handwriting (hand-printing). [Computerwoche84] Computerwoche "Das Penpad-Terminal, zur handschriftlichen Dateneingabe.", Computerwoche, May 4, 1984, Press announcement of PenPad 200 handwriting recognition computer terminal, using pre-defined forms on a digitizer tablet. "Auf das Tablet werden Formulare gelegt, deren Einteilung der Benutzer vorher nach seinen Bedürfnissen festgelegt hat." [ConvisDB85a] Convis, D.B., Grim, P.J., and Reed, M.A. "Retro-stroke Compression and Image Generation of Script and Graphic Data Employing an Information Processing System", United States Patent 4,550,438, October 29, 1985, assigned to IBM Corporation, Armonk, New York. Uses local-maxima chain codes for image compression of script [Cook84] Cook, Mike "Light Pen", The Micro User, Vol. 2 No. 1, March 1984 Light pen - software techniques (via rubber-banding) to get single-pixel resolution from a light pen that only has resolution to character cells in a character CRT display [Crane85a] Crane, H.D., Ostrem, J.S., and Edberg, P.K. "Method for Distinguishing between Complex Character Sets", United States Patent 4,531,231, July 23, 1985, assigned to Communication Intelligence Corporation, Menlo Park, California. Attempt to patent Pencept's "area" definition on Penpad 320: Kanji (Chinese) in one area, Kana/Roman in another (Katakana/Hiragana Japanese) Kanji/Kana by top half / full height of character box (similar to upper/lower case) [Crane85b] Crane, H.D., and Ostrem, J.S. "Complex Pattern Recognition Method and System", United States Patent 4,561,105, December 24, 1985, assigned to Communication Intelligence Corporation, Menlo Park, California. Chinese/Kanji character recognition using stroke shape and curvature Distance metrics between strokes for handwriting recognition of Chinese/Kanji characters Allows for shift in position, rotation of strokes to allow for better recognition matches [DOD85a] United States Department of Defense "Department of Defense Standard: Trusted Computer System Evaluation Criteria, December 1985", DoD STD, December 26, 1985, supersedes CSC-STD , 15 Aug 83 Requirements document for secure computer systems: compare with documentation on Multics for 8:26:23 PM]

106 ( ) History of Pen and Gesture Computing implementation of a secure computer system. See ContentGuard Google. [Davies85] Davies, K. "Continuous Speech Recognition", Probe Research Seminar: The Coming of Age of Voice Recognition, San Francisco California, 1985 Cited in Mangione86: : member of IBM Yorktown Hts. staff (with Tappert) Adaptive recognition dies trying to get complete set of samples [DosterW84a] Doster, W. and Oed, R. "Textbearbeitung auf Personal-Computern mit handschriftlicher Direkteingabe", presentation at Arbeitstagung der Fachgruppe Interaktive Systems der Gesellschaft für Informatik, March 1984, in Zurich, Switzerland. Reprinted in PC-Praxis. Author's address: AEG Aktiengesellschaft, Research Center Ulm, Sedanstrasse 10, D-7900 Ulm, West Germany. Points out their variability model cannot be statistically verified. Describes AEG's general DCR work-station, including driver design for tablet digitizer [DosterW84b] Doster, W. and Oed, R. "Word Processing with On-line Script Recognition", IEEE Micro, October 1984, pp AEG's word-processing user interface for handwriting recognition input. Describes word processing (for handwriting user interface) as two phased: text creation, and text editing. Some use of gestures or recognition macros, separate gesture, pointing (virtual tablet) and text input areas on separate digitizer from display. Compare with PenCept PenPAD and recognition macros, virtual tablet definitions. [DosterW84c] Doster, W. "Das papierlose Büro: Lesemaschine für Handschriften", Funkschau (West Germany), Vol 25, 1984, pp General article on AEG's handwriting recognition system for forms data entry and word processing in an electronic automated office [DosterW85a] Doster, W. "Pattern Recognition Techniques as a Stimulus for Advanced Text Processing", PROTEXT II: Proceedings of the 2nd International Conference on Text Processing Systems, October 23-35, Dublin, Ireland, pp Says putting in multiple variants a deficiency of non-adaptive Says "user-definable" special gesture symbols (for WordStar word processor) better for command, but his examples are of modified standard characters Points out that non-adaptive systems are user-adaptive, and results are the same "perfect" mutual enrollment [DosterW85b] Doster, W. and Schuermann, J. "Bildanalyse von Textdokumenten und Handschriftliche Direkteingabe - Zwei Verfahren Auch für BTX-Anwendungen", Informatik Fachberichte 92, Springer-Verlag, New York, 1985, pp (partial copy) Two research projects at AEG on the automated electronic office: automatic document analysis for O.C.R. images of text and pictures, and on-line handwriting recognition [Downton84] Downton, A.C. and Brooks, C.P. "Automated machine shorthand transcription in commercial applications", Proceedings of INTERACT '84, IFIP Working Group 6.3, London, United Kingdom, September 4-7, 1984, pp Need whole conference proceedings: user interface with handwriting editing. [Dye84] Dye, R., Newell, A.F., and Arnott, J.L. "An adaptive editor for shorthand transcription systems", Proceedings of INTERACT '84, IFIP Working Group 6.3, London, United Kingdom, September 4-7, 1984, pp Need whole conference proceedings: user interface with handwriting editing. van Raamsdonk? [EdgarAD84a] Edgar, Albert D. "Apparatus and Method for Remote Displaying and Sensing of Information using Shadow Parallax", United States Patent 4,468,694, August 28, 1984 Optical shadow parallax pointing input combined with display (so no registration problems). Direct pointing to documents and objects on desk surface. By using multiple shadows or multiple cameras, can extract height information from tablet image, to determine touch and lift. Compare with Wellner? [Ejiri85] Ejiri, Koichi "Method of Assigning Direction Code to Boundary Picture Element in Character Recognition System", United States Patent 4.524,454 Line-thinning, extraction of plausible writing direction from edges of scanned OCR images of characters, then to encode as direction codes / chain codes (There is another paper on this sort of thing in here someplace...) [ElGamalT85a] El Gamal, Taher "A Public Key Cryptosystem and a Signature Scheme Based on Discrete Logarithms", IEEE Trans. on Information Theory, July 1985, vol. IT-31 no. 4 pp Public-key PKI signature algorithm, includes section on mathematical attacks. [Epson84a] (unknown) "Epson RC-20 wristwatch", homepage.mac.com, fetched 2010, posted 1998 (in Japanese, machine translation in file) Touchscreen wristwatch, compared to Casio VDB-1000 databank wristwatch with touchscreen. [Ewing85] Ewing, J., Mehrabanzad, S., Sheck, S., Ostroff, D. and Shneiderman, B. "An Experimental Comparison of a Mouse and Arrow-Jump Keys for an Interactive Encyclopedia", to appear in International Journal of Man- Machine Studies, September, Author's address: Department of Computer Science, University of Maryland, 8:26:23 PM]

107 ( ) History of Pen and Gesture Computing College Park, Maryland Pre-publication version only User-interface on pointing methods: applies to mouse emulation on a tablet? [Flurry85] Flurry, G.A. "Electronic Handwriting Facility", IBM Technical Disclosure Bulletin, Vol 27 No 9, February 1985, pp Description of IBM's handwriting work-station? [FoleyJD84] Foley, J.D., Wallace, V.L., and Chan, P. "The Human Factors of Computer Graphics Interaction Techniques", IEEE Computer Graphics and Applications, Vol 4 No 11, November 1984 The granddaddy of all user-interface papers for interactive graphics [FormsAutomation85] Forms magazine "One-step data entry system links forms with computers", forms automation product review, April, 1985, pp Published by National Business Forms Association, 443 East Monroe Avenue, Alexandria VA Description of DataPad/ AnnoGraphics / Portable Computer handwriting recognition product: was marketed under several different name. [Fox84] Fox, A.S., Kim, J. and Tappert, C.C. "Segmenter for Known Number of Characters", IBM Technical Disclosure Bulletin, Vol 27 No 7A, December 1984, pp Segmentation when writing prototypes, because number of shapes is known [FushimotoH85a] Fushimoto, Hideo; Toyomura, Shigeru; Nishimuro, Yshiaki; Asakura, Osamu; and Rachi, Masahiro "Foldable Electronic Apparatus", United States Patent 4,517,660, May 14, 1985 Foldable calculator/pda, side-by-side configuration, or close and use calculator keys on outside. Keyboard/keypad input, but full display. [GSS84] GSS "GSS-Drivers User's Guide", Second Edition, Graphic Software Systems, Incorporated, March 1984 [Gaines84] Gaines, B.R., McKellar, I.D., Dinger, W.P., and Fast, S.R. "Some Experience in the Real-time Processing of Handwriting", Proceedings of the 7th International Pattern Recognition Conference, Montreal, 1984, pp Says natural user-interface is speech and handwriting together Says tablet digitizers with 1000x1000 resolution and 1% linearity good enough for handwriting (!) Say peak handwriting speed/velocity is 50 inches/second (?), normal is 2.5"/sec Says "skilled" user gets 97%-100%, "unskilled" only 70% recognition success [Genter85] Genter, R.E. "Guidelines for C Program Development", Internal report, Intermetrics, Incorporated, Cambridge, Massachusetts, September 1985 Programming/coding style [Gharachorloo85] Gharachorloo, N. "On-line Character Recognition Using Closed-Loop Detector", United States Patent 4,495,646, January 22, 1985 Tablet-free stylus using mechanical design to detect rough stylus direction of motion Claims handwriting recognition using mechanical sensing of chain codes [GlennWE84a] Glenn, W.E. "Apparatus for Determining Position and Writing Pressure", United States Patent 4,488,000, December 11, 1984, assigned to New York Institute of Technology, Old Westbury, New York. Force/pressure-sensitive digitizer tablet using acoustic sound waves in a sheet [GoldbergA84] Goldberg, Adele "Smalltalk-80: The Interactive Programming Environment", Addison-Wesley, Reading MA, 1984 Introduction to Smalltalk language and environment. (Electronic file.) [Goldwasser85] Goldwasser, Eric, and Goldwasser, Dorothy "Method of Creating Text Using a Computer", United States Patent 4,559,598, December Pen/stylus or other pointing means, select words from dynamic menus to input text. Intended as aid to the blind: compare with QwikWriting and other means for one-handed input of handwritten text with constrained recognition. [Gould84] Gould, J.D. and Alfaro, L. "Revising Documents with Text Editors, Handwriting-Recognition Systems, and Speech-Recognition Systems", Human Factors, Vol 26 No 4, August 1984, pp Applications: text editing much faster with handwriting than with keyboard, even though same number of characters input User-interface: differences in external feedback greatly affect operational speed Userinterface: time spent inputting a command much less than think time (and typing adds to effective think time) User-interface: electronic ink would eliminate spatial displacement within written copy and screen copy Speech simulation of handwriting editing, but no standard proof-editing marks Speech: no visual feedback a minus on speech recognition test Speech: USERs make more errors in speech than with writing or typing Martin89 cites as saying most of time in producing a letter is think time, so user-interface of speech recognition or typing not that great a difference [Graham84a] Graham, Martin "Signature Verification Sensor", United States Patent 4,475,235, October 2, 1984, assigned to Rolm Corporation, Santa Clara, California. Capacitive sheet/plate digitizer tablet [GreenM85a] Green, Mark "The University of Alberta User Interface Management System / UIMS", Proc. 8:26:23 PM]

108 ( ) History of Pen and Gesture Computing SIGGRAPH '85, Vol 19, No 3, 1985, San Francisco July 22-26, 1985, pp UIMS based on Seeheim model: Presentation layer, Dialog Control (Interaction) layer, Application Interface Layer, with user interface / user input events and event handlers. [GreeneR85a] Greene, Richard "The Drawing Prism: A Versatile Graphic Input Device", Proc. SIGGRAPH '85, Vol 19, No 3, 1985, San Francisco July 22-26, 1985, pp Drawing prism: Touch surface/tablet system using total internal reflection and refraction. User marks with brush or any object on one side of a prism, video reads points of contact by viewing at an angle that only sees light reflection where surface it touched. Size up to 16" by 20" constructed in prototype from oil in plastic surfaces. Refers to multiple fingers in finger painting: multi-touch. Cites to Mueller patent using same principle. [HagenMD85a] Hagen, Mark D.; Mendel, Peter J.; Regehr, John L. "Rotatable Display Work Station", United States Patent 4,542,377, September 17, 1985 CRT display that can be rotated mechanically between landscape and portrait orientations. Algorithm for rotate control logic to read characters out from memory for current orientation, with user scrolling for data off edge of display. [Hayamizu84] Hayamizu, S. and Oka, R. "Experimental Studies on the Connected-Words Recognition using Continuous dynamic Programming", Denshi Tsushin Gakkai Ronbunshi, Vol 67-D No 6, June 1984, pp (in Japanese), translated in Systems, Computers and Controls, Vol 15 No 4, 1984, pp Speech recognition: performance goes down with vocabulary size (everybody should show performance with increasing vocabulary size) Recognition errors are confusion (substitution), missing, and "ghost word" [HertzfeldA84a] Hertzfeld, Andy "Folklore - The Original Macintosh: Thunderscan scanner", folklore.org, June 1984: fetched 2015 Modification to early C. Itoh / Apple Macintosh printer to add optical sensing element, high-quality scanner. With large images, added inertial scrolling with coasting at variable speed, hysteresis. Inertial scrolling. [Higgins84] Higgins, C.A. and Whitrow, R. "On-line Cursive Script Recognition", Proceedings of Interact '84, 1st IFIPS Conference on Human-Computer Interaction, September 1984, pp Unlikely that "unconstrained" can be done, since humans find stuff illegible Segmentation at turning points, not Y-maxima/minima Features must have enough variety to be effective, but not be too sensitive Extra feature tests ("optional") for specific characters Context - "higher shape templates" for combined shapes Refers to including new shapes on the fly for mis-recognition Training is "manual: results in inaccurate feature values for templates" (?) From incorrect weighting, best word match does not always come out on top [Hitachi84] Hitachi, Limited: "Humanification product literature", (in Japanese), 1984 Hitachi Kanji/Chinese handwriting recognition product [HondaK85a] Honda, Kazuhiro and Kuramochi, Hiroshi "Pressure-sensitive and conductive rubber", United States Patent 4,533,604, August 6, 1985 Conductive rubber, pressure-sensitive: force on rubber causes carbon, metallic, or other particles to be pushed together, reduces resistance. Type of rubber (elastomeric silicone, etc.) with different density or specific gravity, particles tend to sediment [Honeywell85a] Honeywell Information Systems "MULTICS Project Administrator's Manual", Honeywell Information Systems AK51-02 February 1985 Multics sysadmin manual: as_who for sudo superuser operation, uses Multics ACLs access control lists [HouseVD84a] House, V. Dean "Touch controlled membrane for multi-axis voltage selection", United States Patent 4,444,998 two-sheet resistive film digitizer tablet, measure DC voltage. Also two resistive films on both sides of a conductive sheet. Compare with Krey digital pad? [Huang85] Huang, J.K. "The input and output of Chinese and Japanese characters", IEEE Computer, Vol 18, January 1985, pp ,000 total Chinese characters: three-corner code method TCCM for encoding and indexing for keyboard input [Hulls85] Hulls, L.R. "Absolute Position Coordinate Determining Device employing a Single Phase Difference Measurement to Determine Cursor Position", United States Patent 4,552,991, assigned to Numonics Corporation, Lansdale Pennsylvania, November 12, 1985 Phase-shifting tablet digitizer (electromagnetic?), using not a simple grid, but in each ordinate a mixed set of loops at physically different phase. [HutchinsEL85a] Hutchins, Edwin L; Hollan, James D.; and Norman, Donald A. "Direct Manipulation Interfaces", Human-Computer Interaction, 1985, vol 1, pp Tutorial on direct-manipulation user interfaces circa Early examples of direct manipulation include Sutherland's Sketchpad 1963, ThingLab 1979, electronic spreadsheet programs early 1980's. [IBM85] Anonymous "Multi-Segment System for Recognizing Cursive Writing", IBM Technical Disclosure Bulletin, 8:26:23 PM]

109 ( ) History of Pen and Gesture Computing Vol 17 No 11, April 1985, pp Segmentation algorithm for script (Tappert?) for run-on and script handwriting [IchikawaS84] Ichikawa, S. and Gyoba, J. "Methodological Problems on Pattern Psychophysics", (Japanese journal), Vol 27 No 2, 1984, pp Not sure of English name of Japanese journal Similar to functional attributes, describes general problem of figuring out human recognition mechanisms for multiple features [IkedaE84a] Ikeda, Eiichi; Ikuo, Kayama; and Izaki, Masahiro "Pressure-sensitive conductive elastic sheet", United States Patent 4,448,837, May 15, 1984 Conductive silicone rubber or elastomeric sheet used in touch digitizer: particles are ferromagnetic, coarse particles aligned by magnetic field perpendicular to sheet during fabrication, fine particles near surface. Gives better palm rejection (large contact area, light pressure does not cause conduction?) by making dispersal of particles more uniform (fewer shorts?): responds to sharp pressure of stylus/pen point, but not broad dispersed pressure of heel of user's hand. [Ikeuchi84] Ikeuchi, K. "A Model of Character Recognition by Humans", Proceedings of the 7th International Pattern Recognition Conference, Montreal, 1974, pp Three stages of human recognition: feature extractor, naming, decision Human features: straight line, turned line, end point, curved line Compare with Shillman [Jansen84] Jansen H. et al "Interpretation of freehand drawings for mechanical design processes", Computers and Graphics, Vol 8 No 4, pages , 1984: Pergamon Press Ltd, USA [Jelinek85] Jelinek, F. "The Development of an Experimental Discrete Dictation Recognizer", Proceedings of IEEE, Vol 73 No 11, November 1985, pp Speech: gives results of (huge) number of samples needed to get statistically useful vocabulary coverage Speech: interface problem for word processing is not a trivial one Speech: editing (pointing) with pencilfundamentally better user interface than voice/speech recognition Speech: for editing, best interface will include digitizer and handwritten character recognition for editing marks (mode separation? ref. Buxton "tension") Speech: an adaptive system that does worse with non-native speakers of English (!) (Features are biased? Not really a general adaptive system?) Adaptive speech: cites user who did worse when analyzing own pronunciation (ref: problems of user understanding enrollment problems) [JohnstoneE85a] Johnstone, Eric "The Rolky: A Poly-touch Controller for Electronic Music", Proc. ICMC '85, pp Multi-touch digitizer using total internal reflection on glass: half-silvered mirror to CRT display so that hand does not obscure image. [KADesign85] KA Design group "business plan", March 1, 1985 Puck/pointer: proposed mouse-like tablet product using a small joystick. Contains patent references to Victor B. Kley, 4,435,616, Graphical Data Entry Apparatus, resistive sheet tablet digitizer. [Kahan85] Kahan, S. "Problems in recognizing hand-printed characters", M.S. Thesis, Department of Electrical Engineering and Computer Science, University of California at Berkeley, February 1985 Cited in Kahan87 [Kai-tungHuang85] Kai-tung Huang, J. "The Input and Output of Chinese and Japanese Characters", IEEE Computer, Vol 18 No 1, January, 1985, pp Chinese and Japanese Kanji input via keyboard [Karat84] Karat, J., McDonald, J., and Anderson, M. "A comparison of selection techniques: touch panel, mouse, and keyboard", Technical report No TR , IBM Corporation, Entry Systems Division, Austin, Texas, 1984 Mack89, user-interface on integrated tablet/display [Kasanuki85] Kasanuki et al "Sophisticated Man-Machine Systems (2)", 30th Conference of the Data Processing Society of Japan, pp , Japan Cited in Sklarew patents [KasdayLR84a] Kasday, Leonard, R. "Touch Position Sensitive Surface", United States Patent 4,484,179, November 20, 1984 Membrane-based touchscreen digitizer, top film presses on lower glass causing total internal reflection (compare with FTIR?) detected optically. Can sense multi-touch multiple touches, and sense force/pressure (not clear how). Light detectors are at edges of screen, can be transparent. One use is re-labelable keys (compare with Ken Knowlton, Kaplow) Describes two-finger gestures for rotation. States that device can detect more than one touch at a time by using timing information, and measure force by greater contact area causing stronger signal. [Kataoka84] Kataoka, Hiroyuki and Arai, Yoshio "Character Shaping Device", United States Patent 4,440,513 assigned to Fuji Xerox Co., Ltd. Tokyo Japan, April 3, 1984 Handwriting text editing: Normalizes size, scale, position, shape of handwritten characters prior to printing to be neatly spaced and lined up better, but does not talk about recognition per se [KernighanBW84a] Kernighan, Brian W. and Pike, Rob "The UNIX Programming Environment", Bell Telephone 8:26:23 PM]

110 ( ) History of Pen and Gesture Computing Laboratories, 1984 Original Unix system documentation from Bell Labs: basic shell, troff, sed, system calls for file system, processes, signals, interrupts. Explanation of standard I/O (stdout, stdin, etc.), UNIX-style file permissions (owner, group, world). [KimJ84] Kim, J. and Tappert, C.C. "Handwriting Recognition Accuracy versus Tablet Resolution and Sampling Rate", IEEE 7th International Conference on Pattern Recognition, 1984, Montreal Canada, pp A "nothing" paper: recognition goes down with worse tablet resolution [KimuraS84a] Kimura, Shigeru "Mechanically Feedbacked Touch Sensor for Electronic Painting by Skilled People", Proc. Computer Graphics Tokyo '84, pp Translated from Japanese. Haptic feedback stylus for electronic painting on a tablet: electromagnetic linear actuator in pen/stylus. One use is inputting "ambiguous" values in lighter electronic ink, final values later. [KoalaPad84a] Koala Technologies "Koala Pad: TouchTablet Owner's Manual", Koala Technologies, 1984 Finger/stylus touchpad for AppleII: Low resolution, plugs into joystick port. See also Thornburg. [Konneker84] Konneker, L.K. "A Graphic Interaction Technique Which Uses Gestures", IEEE Proceedings of First International Conference on Office Automation, page 51-55, 1984 Cited in Rhyne86, Buxton87 Rhyne86 cites this as an interesting handwriting input system: Also Goodisman91 Comments that "'C' code is not necessarily portable" [KrasnerG84] Krasner, Glenn, editor "Smalltalk-80: Bits of History, Words of Advice", Addison-Wesley, Reading MA, 1984 Background of Smalltalk language and environment. (Electronic file.) [Krieger85] Krieger, Mark "Writing Portable UNIX Software", talk given at the 1985 UNIFORUM conference in Dallas, Texas by Mark Krieger, January, Speaker's address: UniPress Software Incorporated, Suite 312, 2025 Lincoln Highway, Edison NJ [KruegerMW85a] Krueger, Myron W.; Gionfriddo, Thomas; and Hinrichsen, Katrin "VIDEOPLACE - An Artitifical Reality", Proc. CHI '85, April 1985, pp Interactive environment sensing outlines of user images in video sensing, video display. Shows two-hand (multi-touch) gestures for stretch, resize, move of interactive objects. Compare with interactive desk. Artificial reality / responsive environment. [KuklinskiT84a] Kuklinski, T. "Components of Hand-Print Style Variability", Proceedings of the IEEE 7th International Conference on Pattern Recognition, pp , 1984 Handwriting variability classifiably affected by schooling, country of origin (education), age (small children, elderly), profession (architects print neatly), haste/type of note taking, writing instrument and surface, etc. Early form of generative model for handwriting/handprinting variability: See Ward/Kulklinski [KuklinskiT85a] Kuklinski, T. "A Case for Digitizer Tablets", Computer Graphics World, May, 1985, pp Overview of digitizer tablets: electromagnetic vs. resistive vs. electrostatic, pressure/force measurement. GKS standard supports locator (point), stroke (path), valuator, selector, string (handwriting recognition). Tablet vs. mouse: tablet does not require screen feedback, can do absolute *or* relative positioning/movement. Tablet templates / soft function areas for command input. Future screen inking /electronic ink, portable tablet clipboards [KuklinskiT85b] Kuklinski, Theodore) "Internal speaking notes, Development of Pencept Digitizing Tablet", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1985 History of digitizer performance and technology problems that lead to development of Pencept electromagnetic digitizing tablet with two-coil sensing stylus to correct for stylus tilt. Credits to Dr. Barry Blesser. Contains discussion of performance problems of digitizing tablets in general. [Kurta85a] Kurta Corporation "Kurta IS/PenMouse digitizing tablet, ca. 1985", Kurta Corporation (physical device) PenMouse is a digitizing tablet with a mouse-emulation mode. (Stylus missing). [Kushnir85] Kushnir, M., Abe, K., and Matsumoto, K. "Recognition of Handprinted Hebrew Characters using Features Selected in the Hough Transform Space", Pattern Recognition, Vol 18 No 2, 1985, pp Features are stroke direction, concavities, endpoints, intersections, etc. Says high dimensionality of feature space means template matching takes up lots of memory and compute time Handwriting recognition requires new strategies beyond printed recognition Accuracy test uses larger training set than test set Statistics fail by grouping similar characters, which must be solved by heuristics (!) Issues of test samples versus training samples Source of errors is similar characters: boundary? [Kuzunki85] Kuzunki, S. et al "Handwritten Characters and Graphics Recognition Technology for Workstations", Hitachi Hyoron, Vol 67 No 3, pp , 1985 Cited in Sklarew patents [Lamarche84] Lamarche, F. and Plamondon, R. "Segmentation and Feature Extraction of Handwritten Signature Patterns", Proceedings of the 7th International Conference on Pattern Recognition, 1984, pp Refers to time-axis distortion (variability) in signatures [Lamb84] Lamb, M.R. and Buckley, V. "New techniques for gesture-based dialogue", Proceedings of INTERACT 8:26:23 PM]

111 ( ) History of Pen and Gesture Computing '84, IFIP Working Group 6.3, London, United Kingdom, September 4-7, 1984, pp Need whole conference proceedings: user interface with handwriting editing. [LamportL84a] Lamport, Leslie "Recursive Compiling and Programming Environments: Summary", Self-published manuscript July 1984 https://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html#recursive-compiling Selfmodifying compile construct added to Pascal: can compile and then execute source code, a la Lisp: build environments / development environment. Mentions that the original version of EMACS was written in TECO, not in Lisp -- original Teco editor from Digital Equipment Corporation? [LeeSK84a] Lee, Seonkyoo "A Fast Multiple-Touch-sensitive Input Device", Master's Thesis, Dept. of EE, U. of Toronto, October 1984: 115 pages (partial copy) Capacitive digitizer touchscreen: multi-touch sensor resolution is 64 by 32, increased by interpolation. [LeeSK85] Lee, S.K., Buxton, W. and Smith, K.C. "A Multi-Touch Three Dimensional Touch-sensitive Tablet", CHI'85 Proceedings, pp 21-ff, ACM /85/004/0021 Capacitive digitizer technology, used for sensing multiple touch points of user's finger on surface: uses recursive division to determine position: used to emulate a virtual piano keyboard. Points to problems of delay between gesture and response to gesture. Also: Electronic video file of demonstration at CHI '85. [Leedham84] Leedham, C.G., Downton, A.C., Brooks, C.P. and Newell, A.F. "On-line acquisition of Pitman's handwritten shorthand as a means of rapid data entry", Proceedings of INTERACT '84, IFIP Working Group 6.3, London, United Kingdom, September 4-7, 1984, pp Need whole conference proceedings: user interface with handwriting editing. [LelerW85a] Leler, Wm and Soiffer, Neil "An Interactive Graphical Interface for Reduce", Proc. AAECC-3 Int'l. Conf. on Applied Algebra, Algebraic Algorithms, Symbolic Computation, and Error Correcting Codes, Grenoble France, July , 1985 (extended version of paper) Editing of 2D mathematical expressions using mouse or tablet (symbolic algebra program e.g. Macsyma). Circle gesture subject to problems of selecting (stray) parts of different sub-expressions. [LevyHM84a] Levy, Henry M. "Capability-Based Computer Systems", Digital Press (Digital Equipment Corporation), 1984 General reference/textbook on capability-based computer systems: single address format for all objects: hardware/software, primary/secondary memory. Capability address is a token specifying both ID for object, and access rights. Burroughs B5000/B5500 as early system, supervisor/kernel systems, support for capabilities in object-based design. Multics segmentation. [LewisPH85] Lerner "Peripherals: Astronomy Software", New York Times, May 21, 1985 Mentions PenPad 320 handwriting terminal with handwriting recognition: uses Motorola microchip [Linus84a] Linus Technologies "Business Plan for TechBook Limited Partnership", 5265 Port Royal Road, PO Box 1322, Springfield, Virginia, 22151, February 1984 Original business plan for Linus portable handwriting product: also describes gesture-based user interface [Loken-Dim85] Loken-Kim, K.H. "Artificial Intelligence Techniques and their Application to the Correction of Automatic Speech Recognition Input", Proceedings of Voice I/O Systems Applications Conference, September 10-12, 1985, San Francisco, pp Speech: context correction requires use of linguistic and extra-linguistic knowledge Speech: high recognition rate is not enough: user interface is important Speech: substitution (modal) recognition errors more severe than rejection in performance perception Speech: sequential restrictions in language are called syntax [Lukis85] Lukis, L.J. and Duhig, G.P. "Character Recognition Device", United States Patent 4,493,104, January 8, 1985, assigned to Moore Business Forms, Incorporated, Grand Island, New York. Pressure-sensitive resistive-sheet digitizer. Moore Business Forms DCR patent. [Lyon85] Lyon, Richard F. "Cursor Control Device", United States Patent 4,521,772 June 4, 1985, assigned to Xerox Corporation, Stamford, Connecticut Optical mouse: optical digitizer using passive printed pattern: but recognizes relative motion only, not absolute position. (See other papers in file for additional paper and Web references) [Lyon85a] Lyon, Richard F. "Imaging Array", United States Patent 4,521,773 June 4, 1985, assigned to Xerox Corporation, Stamford, Connecticut Optical mouse sensor, with auto-correlation in analog (?) circuitry. (See other papers in file for additional paper and Web references) [MabusthS85a] Mabusth, Scott "Touch Sensitive Control Device", United States Patent 4,550,221, October 29, 1985 Capacitive grid touchpad: each row electrode and each column electrode measures for capacitance to ground, positions is (interpolated) X for the columns and (interpolated) Y for the row (not capacitance at a crossing point). Row traces and column traces each have extra length where they do not cross over: a capacitance change on a column and not on any row indicates user finger is over/on this extra part, used 8:26:23 PM]

112 ( ) History of Pen and Gesture Computing for additional function buttons. [Mandler85] Mandler, Egerhard, Oed, Richard, and Doster, Wolfgang "Experiments in On-Line Script Recognition", Proceedings of 4th Scandinavian Conference on Image Analysis, Norwegian Institute of Technology, Trondheim, Norway, June 17-20, 1985 Gives many European writing variation styles for "A": features are pathlength and arc; segmentation into single characters without context, not limited to alphanumeric characters, [McCombG85a] McComb, Gordon "Move over keyboard: now it's pens, pads, mice... and more", Popular Science, May 1985, pp. 92-ff. mentions touchscreens (HP-150 computer terminal using infrared light sensors for finger: low resolution; MicroTouch, but does not work with finger), voice recognition, handwriting recognition/pen-computer from Pencept, Chalkboard Powerpad -- multitouch? [Meads85a] Meads, Jon A. "Report on the SIGCHI Workshop on Planning for User Interface Standards", Jon Meads and Associates, 2516 NE 19, Portland, OR 87212, 1985 Discussion of possible UI standards/guidelines. Background includes discussion of multiple device independent layers in conjunction with UIMS: consider virtual devices. [Meads85b] Meads, Jon A. "Friendly or Frivolous?", Datamation, April 1, 1985, pp User-interface user-friendly: users do NOT plan ahead: Fitt's Law says a mouse may be user-friendly, but not a mouse and a keyboard [Meisel85] Meisel, W.S. "Empirical Artificial Intelligence in Speech Recognition", Proceedings of the 1st Annual Artificial Intelligence and Advanced Computer Technology Conference, Long Beach, California, April 30 - May 2, 1985, pp Empirical artificial intelligence: cross between statistical and knowledge-based expert systems analysis Qualitative vs quantitative data/information AI research community has ignored empirical knowledge in expert systems AI: extracting intelligence from empirical evidence vs from human experts AI expert systems: possible flawed assumption is that humans can accurately tell you what they know Curse of dimensionality: your data may just not happen to show a particular feature or interaction among features Sample size vs number of features can be treated formally EAI: using expert knowledge to uncover structures, and using data to verify structures (is not this just normal scientific method?) Reducing "intrinsic dimensionality" of data and features Need off-line (expert?) development of recognition code/tables/features, because too complex to be done in real-time or on-line Human opinion/context has a large role in recognizing/labeling a speech sound Collect speech data for testing from real-world business correspondence and memos linguistic context rules applied to speech recognition [MinskyMR84a] Minsky, Margaret R. "Manipulating Simulated Objects with Real-world Gestures using a Force and Position Sensitive Screen", Computer Graphics Vol 18 No 3, July 1984, pp Touch/pressure-sensitive digitizer tablet: uses four strain gauges / load cells at corners supporting a tablet to sense both position, and strength and direction of force (lateral force), and potential for vision (eye gaze?) gestures. Gestures include tapping, not just shape recognition: note that Penpad also had a "click" gesture of sorts for mouse/character-recognition mode. Includes flicking gesture, using side-force as well as motion. Refers to screen/stylus (finger) parallax problem. Cites to Put-That-There project. Objects can be pushed through virtual "rooms" via doorways at sides of displayed image, display changes to new room. Filter wild data points from touchscreen. Mentions multiple finger touch / multi-touch touchscreen version of hardware. Describes touch screen system using buttons that recognize a "selection" (tap/"hitting") gesture and a "move" (tap and hold/press-and-hold) gesture; the buttons control movement of turtle on the display; the move gesture allows the user to arrange the buttons on the display. I.e. gesture buttons. Has additional gestures, in that one button can be programmed (in various ways, by example) to click on other buttons in a particular order, e.g. to make a "paint something" button. Cutting gesture (similar to rub-out gesture?) instead of command. [Mizzi85] Mizzi, Pierre M.A. "Hand-Held Computer", United States Patent 4,545,023, assigned to Engineering Project Development Limited, London, England, October 1, 1985 Hand-held "slate" style computer with no keyboard, provision of phoneme and/or handwriting recognition, and for TV/Radio reception via plug-in cards [MoriS84] Mori, S., Yamamoto, K., and Yasuda, M. "Research on Machine Recognition of Hand-printed Characters", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-6 No 4, July 1984, pp It's important to know how well DCR system works on low-quality data Recognition of hiragana (small character set) about as difficult as Kanji (large character set) General review of Kanji character recognition Gives figures for total Kanji characters in use in Japan, and in China Mentions earliest commercial Kanji recognition products, ten current Cites a very large data-base collection, which "scope is not sufficient" Cites Blesser Mentions fundamental differences of human vs machine recognition (p 403) Variations unlimited: must be limited for practicality and academic work [MoriS85] Mori, Shunji; Yamada, Hiromitsu; Takeuchi, Kunihiko,; and Do, Michihisa "Feature Extraction System for 8:26:23 PM]

113 ( ) History of Pen and Gesture Computing Digitized Character Information", United States Patent 4,468,808, August 28, 1984 Feature extraction for optical character recognition: compare to chain codes, sub-segments with concavity or convexity, loops: states that convexity/concavity noise is eliminated automatically by processing [Morita85] Morita, M. "Japanese Text Input System", IEEE Computer, Vol 18 No 5, 1985, pp Cited in Kankaanpaa87 special keyboard layout designs for keyboard input for Kanji/Chinese [Murase85] Murase, H., Wakahara, T., and Umeda, M. "Online Writing-Box Free Character String Recognition by Candidate Character Lattice Method", Trans. of the Institute of Electronics Communications Engineers of Japan, Vol J68-D, pp , April (in Japanese). Segmenting strokes into characters using varying spacing: box-less input [MurphyAS84] Murphy, A.S. and Wiseman, J.A. "Optical Digitizing Stylus", IBM Technical Disclosure Bulletin, Vol. 27 No. 5 October 1984, Pages Cited in BennettWE91 Optical stylus digitizer using table with X and Y lines in different colors: appears to work like relative-position optical mouse, not absolute position [Murtagh84] Murtagh, F. and Raftery, A.E. "Fitting Straight Lines to Point Patterns", Pattern Recognition, Vol 17 NO 5, May 1984, pp (abstract only) Feature: finding straight lines in OCR patterns (skeletons?) [MyersBA84a] Myers, Brad A. "The User Interface for Sapphire", IEEE CGA, December 1984, pp Windows management in user interfaces: review of tiled windows, partitioned, overlapping/covered. Active window (accepting input) need not be on top. Pointing devices all treated as relative, not absolute: mouse, electromagnetic tablet, four-button puck on digitizer. Cites to PERQ, Xerox Star, MacIntosh/Lisa. Tracking symbol picture (cursor) shows operation that will be performed on mouse/stylus-up. [NEC84] NEC: "CR-100 product literature", (in Japanese), 1984 Nippon Electric Company 1984 Japanese/Kanji/Chinese on-line handwriting product [Nagayama85] Nagayama, T., Shibuya, J., and Kawakita, T. "Pen-Touch-Type Electro-Magnetic Transparent Touch Panel", Society for Information Display 1985 Digest of Technical Papers, May 1985,pp Transparent digitizer tablet over LCD with DCR [Nagy84] Nagy, G. and Seth, S. "Hierarchical Image Representation with Application to Optically Scanned Documents", Proceedings of the 7th International Conference on Pattern Recognition, Montreal, 1984 Tiling/windows applied for format of written/printed documents [Nagy85] Nagy, G. "Image Database", Image and Vision Computing, Vol 3 No 3, August 1985, pp User-interface aspects of image data bases not given much attention [Naito84] Naito, S. and Masuda, T. "Chinese Character Recognition Based on Personal Handwriting Characteristics", Trans. of IECE of Japan, Vol J67D No 4, April 1984, pp (Abstract only). Science Citation Index Difference in handwriting variations/variability between writers 50% greater than for one writer (?) Claims 96.9% successful recognition when trained to writer, 67.7% for general use without training, for 3256 Kanji categories. [Nandhakumar85] Nandhakumar, N. and Aggarwal, J.K. "The Artificial Intelligence Approach to Pattern Recognition -- A Perspective and an Overview", Pattern Recognition, Vol 18 No 6, pp , 1985 Overview of DCR in the context of artificial intelligence [NationalOA84] National OA: "Panaword RL-W450 / FW-500 product literature", (in Japanese), 1984 National OA 1984 Kanji/Chinese product [Nestor85a] Nestor Inc. "Nestor Inc. Annual Report to Shareholders 1985", Nestor Inc., 122 East 42nd Street, NY NY 01068, 1985 Nestor handwriting recognition company: audit says $500K royalties could not be verified [Nouh84] Nouh, Adnan, Sultan, Abobakr, and Tolba, Roshdi "On Feature Extraction and Selection for Arabic Character Recognition", Arab Gulf Journal of Scientific Research, Vol 2 No 1, 1984, pp Density features for Arabic handwriting recognition: position-by-position matching between characters: Freeman codes as divergence criteria Tilt: 0.01" to 20 degrees, 0.08" between 20 and 40 degrees [Numonics85] Numonics "Numonics #2200 Pad with Low Profile and High Resolution", Numonics Incorporated, 418 Pierce Street, Lansdale PA 19446, 1985 large-format (20" x 20") electromagnetic tablet with stylus or puck. [OKI84] OKI Corporation "SR 220 / SR 230 Online product information", OKI Corporation, 1984 Japanese handwriting character recognition product [OedR84] Oed, R. and Doster, W. "The Intelligent Graphics Tablet - More Than a Mouse", Proceedings of Advances in Microprocessor Peripherals, Journees d'electronique et de Microtechnique, Lausanne, Switzerland, October 8-11, 1984, pp Shows handwriting user-interface with control/editing area much like Penpad or C.I.C. Handwriter Shows diagram for stroke/character segmentation/parsing using overlapping rectangular extents User interface for handwriting into existing applications using keyboard buffer [OedR85] Oed, R. and Doster, W. "On-line Script Recognition - A User-friendly Man-Machine Interface", 8:26:23 PM]

114 ( ) History of Pen and Gesture Computing Proceedings of the COMPINT 85 Conference on Computer-Aided Technologies, IEEE Catalog No 85CH2136-0, pp Shows diagram for stroke/character segmentation/parsing using surrounding rectangles. Separate writing areas on digitizer tablet for text input, gestures, virtual tablet/mouse input: compare with virtual tablets and recognition macros in Pencept PenPad. Makes statement that user-definable trainable/adaptable gesture-symbol recognition simplifies user interface, but this is questionable: semantics of symbol must still be defined, and mnemonicity and non-ambiguity (to recognizer) of a gesture is an open question. [OlsenDR84] Olsen, Dan. R. Jr., Buxton, William, Ehrich, Roger, Kasik, David J., Rhyne, James R., and Siberg, John "A Context for User Interface Management", IEEE Computer Graphics and Applications, December 1984, pp Early article on features needed in UIMS User interface management systems: uniformity across devices and across application. [Omron84] Omron Corporation "Handwriting product information", Omron Corporation, 1984 Japanese handwriting character recognition product [Panati84] Panati, Charles "Panati's Browser's Book Of Beginnings", Houghton Mifflin, Publishers, ISBN (hardcopy book) Chapter 6: "Scribble to Shorthand". Invention of Pens and Papyrus, historical reference to Stylus. Tieronian unistroke alphabet in the history of shorthand, 63 B.C., pages Pitman Shorthand 1937, Taylor shorthand, Gregg Shorthand. Braille. [Parks85] "Apparatus for Signature Verification", Parks, John R; Carr, David R; Fox, Peter F. United States Patent 4,495,644, January 22, 1985, assigned to Quest Automation Public Limited Company, Great Britain. Quest Automation Handwriting product: Micropad/Datapad using resistive film, pressure-sensitive digitizer tablet with hand rest Signature verification considering multiple scores of deviations, signature rejected either by a high combined score, or a very high score on one parameter [Parnell84a] Parnell "Digitizing System", United States Patent 4,473,717 Coarse search on grid tablet (electromagnetic/electrostatic) to find strongest conductor line and two neighbors, then fine determination by three-phase interpolation method on those three lines. Compare with Blesser interpolation. [Pavlidis85] Pavlidis, J., and Van Wyk, C.J. "An Automatic Beautifier for Drawings and Illustrations", A.C.M. Computer Graphics, Vol 19 No 3, July 1985, pp Gesture/graphics input heuristics: cite for Pavlidis. Relevance to sketch recognition? [Pencept84a] Pencept "Penpack (TM) Multiplan (TM) User's Guide", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1984 Another word processor for the Penpad from Pencept [Pencept84b] Pencept "Penpack (TM) IBM (R) Personal Editor User's Guide", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1984 Forms data entry for the Penpad from Pencept [Pencept84c] Pencept "Penware (TM) Penform (TM) User's Guide", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1984 Pencept character set: forms data entry software for handwriting recognition using digitizer. [Pencept84d] Pencept "User Manual for Penpad 320", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1984 User manual for Penpad 320 handwriting-recognition tablet for MS-DOS. [Pencept84e] Pencept "Software Tool Kit for the Penpad 320", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1984 (hardcopy book) Handwriting recognition product of MS-DOS computers: includes gestures ("recognition macros"), definable virtual tablets in digitizer firmware. [Pencept84f] Pencept "Penpad (R) 320 Technical Data", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1984 PenCad, gesture / recognition macros for AutoCad and other MS-DOS CAD applications. [Pencept84g] Pencept "IBM PC peripheral accepts hand-printed or drawn input", IEEE CG&A, May 1984, "New Products", page 94 Product announcement for Pencept PenPad 320, hand-writing-recognition terminal for IBM-PCs. Essentially a keyboard replacement with handwriting recognition, combined with a general purpose digitizing tablet. "Application Packs" for recognition macros / gestures using simulated keyboard/mouse input for unmodified Multiplan, VisiCalc, Word Start, Lotus spreadsheet. Included a simulated "mouse" mode, using either mouse input through the driver, or sending keyboard arrow-key input. [Pencept84h] Pencept "Penware Software for the PenPad / PenPad 320 and Application Software", Pencept Inc., 1984 Handwriting recognition tablet: Brochure on Pencept Software Took Kit with Command Template Editor: 8:26:23 PM]

115 ( ) History of Pen and Gesture Computing Pendraw drawing program, PenCad interface to AutoCAD (Zoom/Magnify gesture/recognition macro), etc., Penform business forms data entry. [Pencept84i] Pencept "Typical Character Shapes which are Recognized by the Pencept System", Pencept Inc., 1984 Handwriting recognition tablet: examples of recognizable distorted variability styles. Compare with CIC. [Pencept85a] Pencept "PenCad (TM) Penpad (R) Interface for Freelance (R)", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1985 Pencept gesture recognition additions / Recognition macros for Freelance graphical drawing program [Pencept85b] Pencept "PenCad Penpad Interface for CAD and Graphics, Version 3.4 (R)", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1985 pen-computing CAD interface using recognition macros / gestures, virtual tablets, stylus instead of a puck [Pencept85c] PenCept, Incorporated "Software Control at the Stroke of a Pen", SIGGRAPH Video Review, ACM Press: CHI '86 Videotape of live demonstration during CHI '85 conference showing gesture recognition / recognition macros, handwriting recognition, interfaces to AutoCad CAD and other applications. [Pencept85d] Pencept "Pencept Marketing Materials", Pencept, Incorporated, 39 Green Street, Waltham, Massachusetts 02154, 1985 Marketing and technical materials from PenCept, Inc. showing handwriting recognition and gesture recognition [Pencept85f] PenCept, Incorporated "Software Control at the Stroke of a Pen", SIGGRAPH Video Review, ACM Press: CHI '86. Available (2009) at Videotape of live demonstration during CHI '85 conference showing gesture recognition / recognition macros, handwriting recognition, interfaces to AutoCad CAD and other applications. [PennycookBW85a] Pennycook, Bruce. W. "Computer-Music Interfaces: a Survey", Computing Surveys, Vol 17, No 2, June 1985, pp 267-ff Mentions Groove and NRC music systems using light-pen, three-d wand digitizer musical score-editing systems mentions in bibliography: some use pen/tablet input [PepperW84a] Pepper, William Jr. "Hand-held musical instrument and systems including a man-machine interface apparatus", United States Patent 4,430,917, February 14, 1984 One-dimensional touch panel element: one dimensional variation on touch pad or digitizer for a musical instrument [PetreP85a] Petre, Peter "Speak, Master: Typewriters That Take Dictation", Fortune magazine, January 7, 1985, pp Review of Speech-to-text speech recognition product and research systems (talkwriters) to replace keyboard terminals: IBM Frederick Jelinek large statistical database, Raymond Kurzweil, mentions dictionary compilation and dictionary size for template-based systems, NEC Yasuo Kato, short words hard to differentiate by sound alone, Victor Zue trained himself to read spectrum diagrams of spoken words for feature-based recognition [Photron85] Photron Limited "OSCON Digitizer technical information: SQ-3100 and SQ-4000MKII Specifications and Technical Guide", Photron Limited, 1985 Electromagnetic tablet, transmitter coils in stylus or cursor (puck). [PingryJ84a] Pingry, Julie "Inputting Graphics By Digitizer", Digital Design, april 1984, pp Overview of digitizer tablets, paper scanners, and video digitizers for simple graphics input. Refers to mouse-style cursors (puck). GTCO stylus pressure-sensitive, measures tilt, used for graphics drawing changes, joystick. Electromechanical 3D digitizers. On-tablet menus. Mentions Pencept dynamic (on-line) character recognition as keyboard alternative. [Plamondon85] Plamondon, R. and Lamarche, F. "Modelization of Handwriting: A System Approach", in "Graphonomics: Contemporary Research in Handwriting", Proceedings of the 1986 Graphonomics Conference, Hong Kong, July 8-12, 1985, Henry S.R. Kao, Gerard P. van Galen and Rumjahn Hoosain, editors, North-Holland, publishers, 1986 Analog filter on digitizer output gives "improved accuracy" (!!??!!). Tablet digitizer jitter noise is inch (?) [Polhemus84] Polhemus "3-Space Digitizer Technical Bulletin", 1984, Polhemus Navigation Sciences Division of McDonnel-Douglass Electronics Co., Colchester, Vermont. [Poole84] Poole, Lon "A Tour of the Mac Desktop", MacWorld, Vol 1 No 1, 1984, pp Cited in JohnsonJ89 MacIntosh desktop: icons, pointers, mouse, menus (Apple/system menu), windows. Windows zoom out from icons. Dialog and alert boxes. Drag gesture. [PortableComputer85] Portable Computer Products Company "Data Pad product announcement", April West Main Street, Suite 600, Fairfax VA 22030, (see also Annographics). Mark Spikell, founder. 8:26:23 PM]

116 ( ) History of Pen and Gesture Computing [PorterT84a] Porter, Thomas and Duff, Tom "Compositing Digital Images", ACM Computer Graphics, vol 18 No 3, July 1974 Early tutorial paper on alpha channel mixing factor for putting graphical object or image over a background. Alpha channel value indicates transparency, should be similar resolution to RGB color values. Suggests matte channels. Operators (BitBLT counterparts?) include clear, src, dst, src-over, dst-over, srcin, dst-in, src-out, dst-out, src-atop, dst-atop, xor. [Prentice85a] Prentice, Thomas C. "Methods for testing digitizer performance", Internal Report, Pencept, Incorporated, 1985 Tablet digitizer performance [Prentice85b] Prentice, Thomas C. "Understanding Digitizer Specifications", Internal Report, Pencept, Incorporated, 1985 Tablet digitizer performance [QuayleSF85a] Quayle, Stanley F. and Schlosser, Philip A. "Electrographic System", United States Patent 4,523,654, June 18, 1985 Electrostatic tablet digitizer using ratiometric measurements of induced AC voltage in a resistive sheet, active stylus/puck. Transparent conductor/semi-conductor of indium tin oxide (ITO) or other metal oxide. In one figure: Sheet is shown between two glass layers. Assigned to Scriptel. [Quest84a] Quest Automation "Micropad: The Point-of-Sale System Using Handwritten Data Entry", Quest Automatction / Micropad Inc. quest Micropad, La Grane Il pressure-sensitive (air-gap resistive sheets touchpad) handprint recognition terminal, 40-character display. Applications shown are pre-printed point of sale forms /receipts, and pre-printed exploded view of parts assembly. "Micropad gets write to the point." Also signature capture, signature verification (!). [RaederG85a] Raeder, Georg "A Survey of Current Graphical Programming Techniques", IEEE Computer, August 1985, pp Overview of graphical programming languages, pre-labview. Nassi-Shneiderman diagrams for overview of program control; data-flow graphs vs. control-flow graphs. Also discusses graphical editors: menus making syntax/semantics more discoverable, etc. [RebmanJ85a] Rebman, Jack "Tactile Sensor for an Industrial Robot or the Like", United States Patent 4,521,685, Mune 4, 1985 Tactile X/Y/Z touch sensor using Hall effect: In addition to force/pressure, can also detect shear/transverse forces. Compare with multitouch arrays. [Richards85] Richards, Joh, T, Boies, Stephen J., Gould, John D. "Rapid Prototyping and System Development: Examination of an Interface Toolkit for Voice and Telephony Applications", IBM Research Division Report RC (#51395), 10/8/85, 9 pages. Also Proc. CHI'86, April 1986, pp rapid-prototyping application generator for voice recognition user interfaces: uses tables of utterances and action to take. [RoadsC85] Roads, Curtis "Research in Music and Artificial Intelligence", Computing Surveys, Vol 17, No 2, June 1985, pp 163-ff Overview of artificial intelligence applied to music composition/understanding. Mentions several graphical/pointing device music-editing projects e.g. Buxton (with gestures), especially concerning internal representation of music, user interface support for differing composition strategies (by user) (composer's assistant). Cites to Countess Lovelace for computer composition of music. [RobinsonD84A] Robinson, Dan "SIDEKICK: Whoever wrote this gem understands Murphy's law", InfoWord, 26. Nov 1984, p 61 Sidekick TSR utility for DOS, with Screen dialer from screen of PC: select telephone number in text on screen, it dials [RoseC85a] Rose, Caroline (ed.) "Inside MacIntosh Vol I-II-III", Addison Wesley, 1985 Book version of Inside Macintosh, guide to using and programming MacIntosh. QuickDraw: bitblt routine copybits, includes SrcOr, SrcXor, and notsrcor modes. Masking is by regions equivalent to bitmap masks: region is any number of closed polygons in drawing coordinates (need not be same as bitmap coordinates). (copymask: see 1994 edition) [RoseC85b] Rose, Caroline (ed.) "Inside MacIntosh Vol I-II-III, Promotional Edition", Apple Computer, March 31, 1985 "Telephone book" (informal) version of Inside Macintosh, guide to using and programming MacIntosh. QuickDraw: bitblt routine copybits, includes SrcOr, SrcXor, and notsrcor modes. Masking is by regions equivalent to bitmap masks: region is any number of closed polygons in drawing coordinates (need not be same as bitmap coordinates). (copymask: see 1994 edition) [Rosson84] Rosson, M.B. "Characterizing Freeform Editing Behavior", IBM Research Report RC (Log #47200) 5/23/84, IBM T.J. Watson Research Center, PO Box 218, Yorktown Heights, NY Touch/gesture interfaces and direct manipulation: Boies said 75% of interactive user time is commands, 8:26:23 PM]

117 ( ) History of Pen and Gesture Computing not input: Rosson says 25%. Common mistake in keyboard-based editing mis-typing command names. [Rouse85] Rouse, Nancy E. "Manual Digitizers for CAD", Machine Design, June 20, 1985 lists all major digitizer tablet companies of the day, and their technologies (sonic, magnetic, magnetostrictive, etc.) 2-D: Altek, CalComp, Elographics, CTCO, Hitachi, Houston Instrument, Kurta, Numonics, Science Accessories Corp (SAC), Scriptel, Summagraphics; 3-D: Micro Control Systems, Polhemus Navigation Sciences, Science Accessories Corp [RubinRV85a] Rubin, R.V., Golin, E.J., and Reiss, S.P. "ThinkPad: a Graphical System for Programming by Demonstration", IEEE Software, Vol 2 No 2, pp Interactive graphical interface language for programming: defining nested/hierarchical data structures by creating boxes inside boxes in graphical user interface. [SAC84] SAC "GP-8-3D Grafbar Technical Bulletin", Science Accessories Corporation, 1984, Southport, Connecticut. Acoustic 2-D/3-D digitizer tablet, large active area, refers to "standard digitizer output formats" [SAC85] SAC "GP-10 Two-dimensional Sonic Digitizer", Science Accessories Corporation, 1985(?), Southport, Connecticut. Acoustic 2-D digitizer tablet, large active area, refers to "standard digitizer output formats": Emulation of vendors Kura, Numonics, Summagraphics, Wacom, Hitachi, Houston Instruments, GTCO, Calcomp and SAC GP-7, GP-8, CP-9, GP-9XL. Formats and behavior customizable via the controller, built-in menu in controller [Sakai84] Sakai, T., Odaka, K., and Toida, T. "Several Approaches to Development of On-line Handwritten Character Input Equipment", Proceedings of the 7th International Conference on Pattern Recognition, 1984, pp Boxed input: command-mode characters straddle boxes New text-editing method for keyboard emulation: uses multiple lines "category 1" characters: complete ambiguous due to sampling resolution [Sakoe84] Sakoe, Hiroaki "Pattern Matching Device Operable with Signals of a Compressed Dynamic Range", United States Patent 4,479,236, October 23, 1984, assigned to Nippon Electric Company, Tokyo, Japan. Iterative approximation to distance function between two patterns [Satoh85] Satoh, K., Ooka, A., and Wada, Y. "Character Recognition Apparatus", United States Patent 4,542,526, September 17, 1985, assigned to Sumitomo Electric Industries, Osaka, Japan. Handwriting character recognition using length and slope of line segments, classification to standard strokes Uses 8-direction chain codes for directions of entire stroke segments. Uses distance between strokes to segment strokes into characters [Schaeken84] Schaeken, B., Verschueren, W., Rene de Cotret, Y. and Hermanne, A. "A Hierarchical System for Handwritten Numeral Recognition", Proceedings of the 7th International Conference on Pattern Recognition, 1984, Montreal Canada, pp Says recognition success % available, but does not give it (!) Handwriting recognition product [SchlosserPA84a] Schlosser, Philip A. and Quayle, Stanley F. "Electrographic System and Method", United States Patent 4,456,787, June 26, 1984 Resistive sheet electrostatic/capacitive digitizer with proximity sensing: Scriptel. See notes from Wang project. [Schuermann84a] Schürmann, Jürgen and Doster, Wolfgang "Umschreibende Rechtecke separieren Zeichen", Computerwoche Nr 46, 9 Nov 1984 Article on the various technologies used to move to a "paperless" office: optical scanning to convert paper documents to electronic form, handwriting/digitizers to capture character writing and sketches without paper [ScriptWriter85] Data Entry Systems "ScriptWriter Technical Information", Data Entry Systems, 6767 Madison Pike, Suite 195, Huntsville, Alabama 35806, 1985 Portable handwriting recognition tablet, two-line 40 char each display. (equals 80 char single line). Tablet technology: point-pressure sensing (elographics?) [Scriptel84] Scriptel "New Products: CAD Graphics Tablet", IEEE Communications Magazine, Vol 22 No 4, April 1984, pp Scriptel electrostatic tablet product advertisement: photograph shows transparent front-surface digitizer, suitable for digitizing drawing under the tablet, or frosted surface for rear-projection. Not subject to magnetic biasing (apparent reference to Summagraphics) [Scriptel85a] Scriptel "Transparent Glass digitizer", Scriptel Corporation, product literature Product information: Scriptel SPD-Series electrostatic tablet: transparent glass digitizer -- shown with puck, not with stylus, resistive decoding technology [Scriptel85b] Scriptel "Scriptel Transparent Glass digitizer", archive.computerhistory.org, fetched 2007 Scriptel glass/transparent digitizer tablet, shown with puck: X [SearbyAD85a] Searby, Anthony D. and Walker, Ian C. "Computerized graphics system and method using an 8:26:23 PM]

118 ( ) History of Pen and Gesture Computing electronically synthesized palette", United States Patent 4,524,421, June 18, 1985 User interface for mixing colors for drawing palette on stylus digitizer. Amount of color picked up from palette can be based on pressure/force (spring potentiometer in stylus), or dwell time. Dwell/press-andhold. [ShackelfordRL85a] Shackelford, Russel L. "Gesture-based Communication with Computers: Feasibility Study", National Science Foundation Report NSF/IST-85002, July pages. multipoint (multi-touch) gesture recognition system using Chalkboard Inc. Hand/palm recognition. Gestures based on touches of fingertips: both up/down (contact) and in/out (from palm) for one-handed multitouch typing. "Power Pad" touchpad/tablet resolution of 100 points / square inch (10 points/inch). Contains analysis of performance and engineering problems with pressure-sensitive Power Pad. [Shakunaga84] Shakunaga, T, Kaneko, H., and Yodogawa, E. "Automatic Writer Recognition by Line-based Spectral Resolution of 2nd Order Statistics", Trans. of IECE of Japan, Vol J67D No 7, July 1984, pp (in Japanese: English abstract only) Science Citation Index Like signature verification, but regular text handwriting: identification vs. verification, with 5.2% reject, % false acceptance [Sharp84] Sharp Corporation "Sharp OA Handwriting product information (in Japanese)", Sharp Corporation, 1984 Japanese office/desktop handwriting character recognition product/system with CRT monitor, printer. [Shneiderman85] Shneiderman, B. "Human-Computer Interaction Research at the University of Maryland", unpublished manuscript, October, Author's address: Department of Computer Science, Human-Computer Interaction Laboratory, University of Maryland, College Park, Maryland Later published in SIGCHI Bulletin, January 1986, Vol 17 no 3, pp User-interface studies on menu structure: globally visible menu helps; arrow keys for pointing vs mouse; manuals are better than on-line help (why?); user preference for touch-screen pointing tablet/display. Synchronized scrolling of two lists of dissimilar but (semantically) related data objects. Direct-manipulation touchscreen/tablet interface for DOS: compare with Momenta? [Shridhar84a] Shridhar, M. and Badreldin, A. "A tree classification algorithm for handwritten character recognition", Proceedings of the 7th International Joint Conference on Pattern Recognition, 1984, pp Cited in Kerrick88 Kerrick88 cites as reference on sequential, hierarchical, and tree-based methods for character recognition [Shridhar84b] Shridhar, M. and Badreldin, A. "High accuracy character recognition algorithm using Fourier and topological descriptors", Pattern Recognition, Vol 17, 1984, pp % accuracy on handwriting recognition of numerics: Fourier on outlines confuses 2 and 5, 6 and 9, sometimes 4 and 7 due to rotational invariance, so topological analysis must follow. [Sklarew85] Sklarew, Ralph "Linus Technologies Inc. Historical notes", Historical notes at Write-Top introduced by Linus Technologies in 1987, mentions earlier work (ca 1984) with Robert Nadeau at George Mason University [SlikowskiJ84a] Slikowski, Joseph "Interactive Computer-Aded Design System", United States Patent 4,451,895, May 29, 1984 User interface for a CAD/CAM design system using two screens: one an unobstructed graphics display screen, the other a light-pen or touch-screen menu screen with dynamic menus. Because of targeting problems, operation is on lift of the stylus or lightpen, not on touch-down. [Swan85] Swan, K. "Pragmatics of Programming Voice Recognition Technologies", Proceedings of Voice I/O Systems Conference, San Francisco California, 1985 Cited in Mangione86 Adaptive recognition: "trying to retraining an already overtrained behavior, such as speech, is not recommended" Training/adaptive: users are unlikely to become expert users if they become discouraged naive users Speech: human factors gap: most system designers and users weak in human factors/user interface issues Speech: system must be readily interfaceable to external host applications Speech: user do not want any "control knobs" on their user interface Speech: features for good interface design never all on one product Speech: user interface system designer cannot differentiate between user and machine error Speech: products ignore transducer (microphone) problems, even though critical Speech: writing custom software for applications kills the product Speech: software must be transportable between programmers, not computers Speech: buffering problems cause system to lose second word even though product works fine Speech: speech equivalent of "graphical doodling" [SweetRE85a] Sweet, Richard E. "The Mesa Programming Environment", ACM SIGPLAN Notices, Vol 20 No 7, July 1985, pp Evolution of Mesa/BCPL/Alto debugger and IDE environment into multiple window environment. Threebutton mouse, multi-click gestures to select more, right mouse button adjusts selection (middle button is pop-up context menu). Also mentions lasso gesture (drawing a box around to select) and underlining gesture to select. [SziklaiGC84a] Sziklai, G.C. "Methods and Apparatus for the Automatic Classification of Patterns", United States 8:26:23 PM]

119 ( ) History of Pen and Gesture Computing Patent 4,454,610, June 12, 1984, assigned to Transaction Sciences Corporation, New Orleans, Louisiana. How to find the signature and courtesy amount on a check using a grey-scale image Signature verification via OCR scanning [TappertCC84a] Tappert, C.C. "Dehooking Procedure for Handwriting on a Tablet", IBM Technical Disclosure Bulletin, Vol 27 No 5, October 1984, pp Hook removal: hooks are a problem of the tablet [TappertCC84b] Tappert, C.C. "Delayed Stroke Processor for Handwriting Recognition", IBM Technical Disclosure Bulletin, Vol 26 No 12, May 1984, pp Delayed strokes: aligns dots only with cusps, crosses on down side of cusp Refers to character cut points in cursive character segmentation [TappertCC84c] Tappert, C.C. "Adaptive On-line Handwriting Recognition", Proceedings of the 7th International Conference on Pattern Recognition, 1984, Montreal Canada, pp Refers to context by "alphabet selection" Mentions U/V ambiguity for single writer (see also Jans75) Mentions A/H, h/n, a/u ambiguity as problem of dynamic matching algorithm (!) Ambiguity of 1,l,I, and upper/lower C,K,O,P,S,V,W,X,Y,Z Mentions "new variations" even for single user (!) Mentions sloppy and ambiguous writing as separate cases Mentions "alphabet selection" like our shift boxes Discrete vs continuous writing, but testing was different (biased) Mentions problems with his hook removal algorithm Claims 94% recognition success, 98% if restricted subset, with three familiar writers Says dictionary lookup would give 100% recognition (!) Mentions segmentation for case where text is known (but when is that real?) Testing found several omitted variants from a "good" system [TappertCC85a] Tappert, C.C. "An Adaptive System for Handwriting Recognition", IBM Research Report RC (Log #50249) 5/21/85, 1985, IBM T.J. Watson Research Center, PO Box 218, Yorktown Heights, NY (to appear in Acta Psychologica). Implies need for big data base: "vast variation, need larger quantities" Refers to intra-writer variability Claims at most four strokes in an English (Latin) letter Mentions characters (C,c, 1,I,J, etc.) only distinguishable by context Mentions need to accommodate new variations: "small sample of data" Baseline drift, more writing time means more prototypes [TappertCC85b] Tappert, C.C. et al "Handwriting Recognition on Transparent Tablet over Flat Display", IBM Research Report RC (Log #52695) 3/3/86, 1986, IBM T.J. Watson Research Center, PO Box 218, Yorktown Heights, NY (To appear in 1986 Society for Information Display International Symposium Digest of Technical Papers). Essential to success of character recognition is a natural human interface Character recognition has biggest performance demands on digitizer tablets Refers to commercially available handwriting recognizers, but not by name Mentions some performance problems of commercially available tablets Refers to minimum tablet tick resolution for character recognition Mentions many tablet performance problems, "natural" writing speed [Tarnopolsky84a] Tarnopolsky "Graphic Answer Input Device for a Teaching Machine", United States Patent 3,793,469 Lukis87 on digitizer tablets [TeitelmanW84a] Teitelman, Warren "The Cedar Programming Environment: A Midterm Report and Examination, Report CSL-83-11", Xerox PARC Palo Alto Research Center, Report CSL-83-11, June 1984 Cedar language, OS, and programming system: successor to Smalltalk. Combines several smaller papers. Dorado bitmapped display, on-line documentation. [TeitelmanW84b] Teitelman, Warren "A Tour Through Cedar", Proc. ICSE '84, 7th Intl. Conf. on Software Engineering, pp Cedar language, OS, and programming system: successor to Smalltalk. Shows GUI, Documentation Browser for on-line documentation. Cedar interpreter part of environment: basically saying having a source-level debugger is helpful. Tiled window management. Window==Viewer/View. [ThomasJC85] Thomas, John C. "Human Factors in IBM", IBM Thomas J. Watson Research Center, Report RC (#50753), July 15, 1985 organizational note on human factors group at IBM: mostly in service to product offerings. [ThornburgDD84a] Thornburg, David D. "The KoalaPad Book: Ideas and Activities from the Inventor of the Touch Tablet", Micro Computer Books, 1984 Applications for KoalaPad, small "pressure_sensitive" touch tablet, two control buttons on top edge, 256x256 resolution. Music editing, drawing, electronic music controller. Mentions spikes to upper-left corner (0,0) at light pressure. Resistive tablet? [Tyler84a] Tyler, Mark "Touch Screens: Big Deal or No Deal?", Datamation, Jan 1984, pp (partial copy) Review of Easel system at Chemical Bank: mentions limitations of touchscreens: gorilla-arm/arm-fatigue, low resolution. Shows highlighting of buttons in screen, simulated virtual keyboard (alphabetical) for stock trading. Cited in touch-screen references such as C. Marlin Brown. 8:26:23 PM]

120 ( ) History of Pen and Gesture Computing [UmphressD85a] Umphress, David and Williams, Glen "Identify verification through keyboard characteristics", Int'l. Jnl. of Man-Machine Studies, Vol 23, 1985, pp Biometric authentication of users based on typing patterns: keystroke latency / time between strokes, per key. Can be applied continuously while user is at keyboard. See also Saltzer [VanDamm84] Van Damm, A. "Scientific American", Introduction to special issue on computer technology, September 1984 Apple MACINTOSH user-interface is really neat [VanRaamsdonk84] Van Raamsdonk, Cornelis "Apparatus for Text Editing and Processing", United States Patent 4,475,239, October 2, 1984, assigned to Olympia Werke AG, Wilhelmshaven, West Germany. User-interface for doing word processing with handwriting recognition, handwriting annotation input: prior art to just about every ink-editing system you can imagine. Editing of electronic ink. [Verschueren84] Verschueren, W., Schaeken, B., Rene de Cotret, Y., and Hermanne, A. "Structural Recognition of Handwritten Numerals", Proceedings of IEEE 7th International Conference on Pattern Recognition, Montreal, Canada, July 30-August 2, 1984, pp Cites 6-0 ambiguity for single writer Says 79% accuracy, "then improved", but does not say how much Kerrick88 cites as example system using both locally and globally computed features [VideoPlace85a] Krueger, M.W.; Gionfriddo, T.; and Hinrichsen, K. "VideoPlace Sampler: Experiments in Man Machine Interaction", Artificial Reality Lab, Univ. of Conn., fetched 2012 from YouTube.com One of several videos of Myron Krueger work on VideoPlace, showing multi-hand/multi-touch gestures such as pinch and stretch: e.g. at running time 2:28 and 4:17. Two finger typing. [Wacom85a] WACOM Corporation "Wacom CU-510 Power Supply for Wacom digitizer tablet, ca. 1985", WACOM Corporation (physical device) Just the power supply, check for tablet in attic [Wakahara84a] Wakahara, T. and Umeda, M. "On-line Cursive Script Recognition using Stroke Linkage Rules", Proceedings of IEEE 7th International Conference on Pattern Recognition, Montreal, Canada, July 30-August 2, 1984, pp Not really cursive handwriting recognition: connected strokes in Japanese Kanji writing. Appears to presume stroke order is highly predictable for Japanese/Chinese writing. [Wakahara84b] Wakahara, T., Odaka, K., and Umeda, M. "Electro-Luminescent Display Superposed by Transparent Electromagnetic Coupling Tablet and Its Application to Script Input Japanese Word Processor", (source?) (in Japanese), 1984 Integrated DCR tablet and display, electronic ink Display parallax corrected by user-selectable offset in electronic ink [Waltz85] Waltz, D.L. and Pollack, J.B. "Massively parallel parsing", Cognitive Science, Vol 9, 1985, pp Cited in Feldman88 [WangPSP85] Wang, P.S.P. "Dual Level Pattern Recognition System", United States Patent 4,521,909, June 4, 1985, assigned to Wang Laboratories, Inc., Lowell, Massachusetts. Two-level character recognition: a first cut may or may not work, but a second pass sorts out characters left ambiguous by the first pass Template matching with learning for new patterns Two-level template matching using a coarse digitizing grid, then a fine grid for what's left: only advantage is shorter comparison time [Ward85a] Ward, Jean Renard "Software Control at the Stroke of a Pen", SIGGRAPH Video Review, ACM Press: CHI '86 Videotape of live demonstration during CHI '85 conference showing gesture recognition / recognition macros, handwriting recognition, interfaces to AutoCad CAD and other applications. [Ward85b] Ward, J. and Blesser, B. "Implications of Using Interactive Hand-print Character Recognition for Computer Input", Proceedings of the 1985 COMPINT Trends and Applications in Computer Graphics Conference, May 1985, IEEE Catalog No 85CH Pre-cursor to Ward85 IEEE article: only cite Ward85, not this one [Ward85c] Ward, J. "UNIX as a development tool for a non-unix microprocessor", CommUNIXations, August/September 1985, Vol V No 5, pp 26-30; ACM SIGSOFT Software Engineering Notes Vol 10 No 3, July 1985, pp Cross-platform development for Pencept handwriting recognition tablet processor from Unix to embedded processor, using Unix-compatible kernel. Mostly a detailed report on requirements for embedded firmware development in "C" and Unix. [Ward85d] Ward, J. "Method and Apparatus for Removing Noise at the Ends of a Stroke", United States Patent 4,534,060, August 6, 1985, assigned to Pencept, Incorporated, Waltham, Massachusetts. Eliminate wobbles at ends of strokes for pre-processing handwritten characters [Ward85e] Ward, J. and Nilssen, A. "Apparatus and Method for Emulating Computer Keyboard Input with a Handprint Terminal", United States Patent 4,562,304, assigned to Pencept, Incorporated, Waltham, Massachusetts, 8:26:23 PM]

121 ( ) History of Pen and Gesture Computing December 31, 1985 Keyboard emulation patent intended to blockanybody from doing on-line character recognition with standard keyboard input [Ward85f] Ward, Jean Renard "Pencept's Challenge", /usr/group CommUNIXations, August/September 1985 engineering article on hardware/firmware development effort for the PenPad 300/320 tablet/handwriting products [Ward85g] Ward, J. and Blesser, B. "Interactive Recognition of Hand-printed Characters for Computer Input", IEEE Computer Graphics and Applications, September 1985, pp Survey/review article on problems and techniques associated with Gesture/Handwriting recognition. Mentions digitizer performance and accuracy, ambiguity of training-based recognition (train-the-user / train the recognizer), boxed input, parsing of strokes to characters in text. PenPad handwriting recognition / gesture product from Pencept circa 1985: has a press-and-hold/long-hold operation, using the barrel button. Long-hold button gesture used to change modes. [Watanabe85] Watanabe, Y., Gyoba, J., Hirata, T., and Maruyama, K. "A Psychological Approach to the Human Recognition of Ambiguous Characters", Journal of the Institute of TV Engineers of Japan, Vol 39 No 6, pp , 1985 (in Japanese). Mentions different "correct" labels for readers and writers of characters Doing same stuff as early Shillman/Blesser work, but for Japanese Refers to "confoundable Katakana" characters "Hayashi's Quantification, model II" [Weibel84] Weibel, Peter "Zur Geschichte und Aesthetik der digitalen Kunst", Supplement to catalog of Ars Electronica 84, Linz Austria, September 1984 History of computer graphics as artistic medium (mostly animation), first section a discussion of the technical implementation. States that for graphical artistic input, a data rate of 24 points/second on a digitizing tablet with a stylus is usually enough. [Werth85a] Werth, L.J. and Paulson, L.G. "Pattern Processing System", United States Patent 4,541,115, September 10, 1985, assigned to Pattern Processing Technologies, Minneapolis, Minnesota. Visual image inspection pattern matching using address loop correlation between images: recognition part of a trainable system Visual pattern image recognition: responds to time-varying images (patterns over time: claims application to speech recognition as well) [Werth85b] Werth, L.J. and Paulson, L.G. "Training Controller for Pattern Processing System", United States Patent 4,504,970, March 12, 1985, assigned to Pattern Processing Technologies, Minneapolis, Minnesota. Visual image inspection pattern matching using address loop correlation between images: trainable system [Weyer85] Weyer, S.A. and Borning, A.H. "A prototype electronic encyclopedia", ACM Trans. Office Information Systems, Vol 3 No 1, 1985, pp Prototype of interactive electronic encyclopedia, beyond browsing information hyperlinked based on semantic content, multimedia pages. Physical user interface of prototype is mouse/gui. See also ThingLab. [WhetstoneA84a] Whetstone, Albert L. "Position Determination Device", United States Patent 4,482,784, November 13, 1984 Magnetostrictive digitizer tablet using L/C delay line tuned to match propagation speed of magnetostrictive pulses. [Whetton85] Whetton, Cris, quoted in "A Glitch in Spline Shaved Line", ACM Software Engineering Notes, Vol 10 No 5, pp 9-10 regarding an incident where published curves for a control system were transcribed from photocopies, October, Xerox-copy error, for tablets, optical scanners: "Photocopies and scanners cannot lie: but they sure can distort!" [Wikipedia84a] KoalaPad "KoalaPad", Wikipedia, fetched 2015 Historical information on KoalaPad finger/stylus touchpad for AppleII: Low resolution, plugs into joystick port. See also Thornburg. Compatible with Atari CX77 Touch Tablet, with button on stylus instead of on touchpad. [WolfeAL84a] Wolfe, Andrew Lawrence and Barrett, Gary Lloyd "Multimodal Touch Sensitive Peripheral Device", United States Patent 6,037,930, Marcy 14, 2000 Priority date is Touchpad used to simulate mouse mode, absolute mode, joystick mode, and numeric keypad mode in controller. Compare with Buxton virtual devices, or Pencept virtual tablets and virtual devices? [Wong85] Wong, K.H. and Fallsize, F. "Dynamic Programming in the Recognition of Connected Handwritten Script", Proceedings of IEEE Conference on Artificial Intelligence Applications, 1985, pp Tests limited to only one writer Script recognition: complains about segmentation mismatches Context: says storing templates for all possible words not practical [WuWT84] Wu, W.T. "Heuristic Approach to Chinese-Character Search", Kybernetics (Great Britain), Vol 13 No 1, 1984, pp (abstract only). 8:26:23 PM]

122 ( ) History of Pen and Gesture Computing [Yankelovich85a] Yankelovich, H., Meyrowitz, N., and van Dam, A. "Reading and writing the electronic book", IEEE Computer, Vol 18 No 10, pp 15-30, 1985 Electronic book / e-book: mostly about hypertext linking and (multiple) reader annotations, with multimedia -- not portability. Cited in Barker,PG89 [Ye84] Ye, P.J., Hugli, H., and Pellandini, F. "Techniques for On-line Chinese Character Recognition with Reduced Writing Constraints", Proceedings of the 7th International Conference on Pattern Recognition, Montreal, 1984, pp On Chinese, 99% accuracy if use stroke order/sequence, 92% if do not Test protocol: says extrapolation from 500 to 2500 Chinese characters straightforward [YouZ84] You, Z. and Jain, A.K. "Performance evaluation of shape matching via chord length distribution", Computer Vision and Graphical Information Processing, Vol 29, 1984, pp Cited in Taxt90 [YoungG84a] Young, Gales "Hugh Le Caine's 1948 Sackbut Synthesizer Performance Mode of Electronic Instruments", Proc. ICMC Int'l. Computer Music Conf. 1984, pp Biography and technical details of Hugh Le Caine touch-sensitive electronic musical instruments, Electronic Sackbut 1948, Elmus Labs. Goal was to allow performer to control timbre and other qualities of tone on electronic keyboard instruments. Piano-style keyboard, keys could be deflected left or right to "lip" the pitch. [Zhang84] Zhang, T.Y and Suen, C.Y. "A Fast Parallel Algorithm for Thinning Digital Patterns", CACM, Vol 27 No 3, March 1984, pp Skeletonization/thinning based on opposing corners and boundaries along diagonals, preserving endpoints and pixel connectivity [Zue85] Zue, V.W. "The Use of Speech Knowledge in Automatic Speech Recognition", Proceedings of IEEE, Vol 73 No 11, November 1985, pp General-purpose pattern recognition not extensible to read world of multiple speakers, large vocabulary, continuous speech. Phonetically-based approaches abandoned for adaptive recognition because no one could deal with phoneme detection well. General-purpose pattern recognition appeals because it is simple, but it does not scale up to real problems. Variability in speech due to co-articulation of phonemes. Human reading of spectrograms: speech segmentationbetter on nonsense words than on regular text (!). Knowledge of allophonic variations helpful for phonetic decoding. Use known properties of human auditory system to reduce irrelevant acoustic variabilities. Much recognition to be learned from studying human decoding of speech signal. Synthesis (artificial generative variability) methods no help on analysis/recognition. Stressed syllables are "islands of reliability" in speech segmentation. Recognition dichotomy of template matching vs feature extraction not absolute. Template matching works well on small, distinct set, but not extensible to real world. Speech signal is a special kind of signal. Ignorance: correctly defining what you do not know is critical: tells you what to work on to make the best progress. Ignorance: admit what you do not know, then do something about it. biblio83 (Prev page) Main Page (Next page) biblio86 An accusation is its own truth: the stain will never wash away completely. "Die Hand, die Du nicht abschütteln kannst, musst Du schütteln." -- altes arabisches Sprichwort 8:26:23 PM]

123 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1986 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:30 EDT Mirrors of this page can be found at: (Generally most up-to-date) biblio85 (Prev page) Main Page (Next page) biblio88 [AEGTelefunken86] Richard Oed, Eberhard Mandler, Wolfgang Doster and Juergen Schuerman "Personal correspondence", September 1986 Independent group working on prototype on-line recognition product [Abe86] Abe, K., Azumatani, Y., Mukouda, M., and Suzuki, S. "Discrimination of Symbols, Lines, and Characters in Flowchart Recognition", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Fundamental problem is separating/segmentation (flowchart) symbols drawn small size from characters Problem of broken lines and curves messes up flowchart/character recognition Sketch recognition [AhmedP86a] Ahmed, P. "Computer recognition of totally unconstrained handwritten ZIP codes", Ph.D. Thesis, Department of Computer Science, Concordia University, Montreal, Canada, 1986 OCR static recognition of zip and postal codes (digits only). Hybrid recognition (two layers). Recognition rate of 89%, but substitution rate 10%, rejection rate 5% -- doesn't add up? Secondare back-end recognition for rejected characters, 99% success, but states 7% rejection rate -- does not add up? No statement of segmentation errors, binarization problems. [AhoAV86a] Aho, Alfred V.; Sethi, Ravi; and Ullman, Jeffrey D. "Compilers: Principles, Techniques, and Tools", Addison-Wesley, Reading, Massachusetts 1977 Compiler/programming language design: later edition of the "Dragon Book" [AminA86] Amin, A. and Masini, G. "Machine Recognition of Multi-font Printed Arabic Texts", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Arabic recognition: 95% segmentation, 85% recognition on 100 Arabic words. Commercial product by Doctor Mark Spikell. [AnnoGraphics86a] Spikell, Mark "Personal correspondence", AnnoGraphics / DataPad, 1986 Correspondence concerning presentation to PHIGS/GKS standards on handwriting recognition, presentation at George Mason University. Marketing of DataPad terminal? [AnnoGraphics86b] AnnoGraphics "Datapad (TM) Portable Hand-print Character Recognition Data Entry System", AnnoGraphics, Incorporated, Kelley Driver, Fairfax, VA 22030, 1986, Mark Spikell, founder. 8:26:26 PM]

124 ( ) History of Pen and Gesture Computing Handwriting recognition / data-entry tablet terminal, single line display, paper forms input. [Aoki86] Aoki, K. and Yamaya, Y. "Recognizer with Learning Mechanism for Hand-written Script English Words", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Recognition rate stated as 85.4% with three writers (but says "real" performance would be much better). [Applicon86] Applicon "Bravo3 User's Guide", Applicon Incorporated, Ann Arbor, Michigan 1986 cited in RubinSM84: Ledeen recognizer? [Averbuch86] Averbuch, A., Bahl, L., Bakis, R., Brown, P., Cole, A., Dagget, G., Das, S., Davies, K., DeGennaro, S., de Sourza, P., Epstein, E., Fraleigh, D., Jelinek, F., Katz, S., Lewis, B., Mercer, R., Nada, A., Nahamoo, D., Picheny, M., Shichman, G., and Spinelli, P. "An IBM-PC Based Large-Vocabulary Isolated-Utterance Speech Recognizer", IBM Research Report RC (Log #53271) 1/28/86, IBM T.J. Watson Research Center, PO Box 218, Yorktown Heights, NY Trainable Speech recognition: a gigantic natural data-base of 27 million words in correspondence was still not enough for coverage of actual vocabulary to get adequate performance from use of a trained dictionary. Enrollment: in 27 million words of correspondence, several days of week and months of year missing from 5000 most frequent and had to be added by hand. [Babaguchi86] Babaguchi, N., Tsukamoto, M., and Aibara, T. "Knowledge Aided Character Segmentation from Handwritten Document Image", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Complete character segmentation (in OCR) not possible only with general knowledge. Specific knowledge of characters will help segmentation (i.e. segmentation not general-purpose-method solvable as humans do it?) [BachMJ86a] Bach, Maurice J. "The Design of the Unix Operating System", Bell Telephone Laboratories, 1986 Unix operating system design and programming reference: overview of login implementation, kernel etc. Includes API definitions: chmod for file permissions. no ACL access control list documented. [Bahl86] Bahl, L.R. "Speech Recognition - An IBM Research Project", unpublished manuscript, December Author's address: Continuous Speech Recognition Group, IBM T.J. Watson Research Center, Yorktown Heights, NY Speech recognition: enrollment of 1000 sentences, testing on only 50 sentences [BairdHS86a] Baird, H.S. "Accurate Skew Estimation and the Top-Down Analysis of Document Images", unpublished manuscript, December 12, 1986, submitted to 1987 IEEE/CS 1st International Conference on Computer Vision, London. Author's address: AT&T Bell Laboratories, 2C-557, 600 Mountain Avenue, Murray Hill, NJ Published version? Automatic rotation/scaling/skew-angle correction using "minimum energy" in OCR preprocessing. [BairdHS86b] Baird, H.S., Kahan, S., and Pavlidis, Theo "Components of an Omnifont Page Reader", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Segmentation: Accidental merging/splitting of characters known to be a severe problem in optical character recognition. Claims that production of multi-font OCR recognition an engineering technology problem, not science [Bao-chang86] Bao-chang, P. "Floating Mask Method for Extracting Hand-printed Character Features", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Chinese OCR: 99.93% accuracy, 200,000 samples of 51 characters by 90 persons Testing: uses very small subset of Chinese, thereby avoiding the problem of large character set. [BarkerPG86] Barker, P.G., Najah, M., and Manji, K.A. "Pictorial communication with Computers", manuscript in preparation, October, Interactive Systems Research Group, Department of Computer Science, Teeside Polytechnic, County Cleveland, United Kingdom. Pictorial user interfaces (icons): user interface varies dynamically -- changes in response to user actions appropriately. Example shows system with digitizer/tablet, static paper overlays (templates), pecking/pointing to printed icons/pictures. Character stroking/handprint terminal with handwriting recognition. [BierEA86a] Bier, Eric Allen and Stone, Maureen C. "Snap-Dragging", Proc. SIGGRAPH '86, Dallas, Texas, Aug 1986, SIGGRAPH Vol 20 No 4, 1986, pp Snap to grid vs. heuristic constraints similar to compass and ruler for pointing and positioning, e.g. in Gargoyle technical illustration system. Gesture of touching two points of a polygon established constraint of horizontal or vertical. [Bin86] Bin, L. and Shuxiang, Z. "A New Approach to Recognition of Both Handwritten and Multi-font Printed Chinese Characters", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Model for recognition taken from Chinese education from psychologists (cognitive?), educators, linguists. The better a method fits a (cognitive) model, the better the recognition algorithm is 8:26:26 PM]

125 ( ) History of Pen and Gesture Computing [BlesserB86a] Blesser, B. "Digitizing Tablet System Having Stylus Tilt Correction", United States Patent 4,577,057, assigned to Pencept, Incorporated, Waltham, Massachusetts, March 18, 1986 Two-coil digitizer tablet stylus/pen for tilt measurement and/or error correction [BlesserB86b] Blesser, B. "Digitizing Tablet System Including a Tablet Having a Grid Structure Made of Two Orthogonal Sets of Parallel Uniformly Sized and Spaced U Shaped Loops of Conductive Material", United States Patent 4,582,955, assigned to Pencept, Incorporated, Waltham, Massachusetts, April 15, 1986 Digitizer tablet geometry to minimize stray signal pick-up [BlesserB86c] Blesser, B., and Ward, J. "Human Factors Affecting the Problem of Machine Recognition of Handprinted Text", Vol III Proceedings of the 7th Annual Computer Graphics Conference, National Computer Graphics Association, May , pp Paper shows all the variations of "G"s: actual presentation did "A"s. Gives many examples of combinatorial explosion of handwritten character variability from base forms. [BlonderGE86a] Blonder, Greg E.; Lamola, Angelo A.; and Lieberman, Robert A. "Variable Gap Device and Method of Manufacture", United States Patent 4,617,608, October 14, 1986 Variable-gap capacitive pressure sensor (self-capacitive) using compressible rubber: transparent materials suitable for touchscreen. [BonoPR86] Bono, Peter R. "Support for Character Recognition Devices by the Graphics Standards", presentation to PHIGS graphics-standards committee, May 23, 1986 Presentation on behalf of several on-line pen computing companies to the ANSI graphics standards committees: organized by Pencept, includes CIC, IBM others. Request concerned defining handwriting input for user interfaces (characters with position: like gestures) in some fashion, rather than having separate locator and text input. [Borning86] Borning, Alan "Graphically Defining New Building Blocks in ThingLab", Human-Computer Interaction, 1986, Vol 2 No 4 pp (abstract only) Animation editor in ThingLab, with object constraints [BraultJJ86] Brault, J.J. and Plamondon, R. "Coupling Visual and Dynamic Features to Study Handwritten Signatures", Proceedings of Graphics Interface/Vision Interface 86, Vancouver, B.C May 1986 Shows defective tablet line, but thinks it's just digitizer noise. [Burr86] Burr, D.J. "A Neural Network Digit Recognizer", Proceedings of the 1986 International conference on Systems, Man, and Cybernetics, Atlanta, Georgia, 1986, pp Martin,GL90 [Buxton86] Buxton, W. "Chunking and Phrasing and the Design of Human-Computer Dialogues", Proceedings of IFIP World Computer Congress, Dublin Ireland, September 1-5, 1986; also Baecker95a Towards the Year 2000 Gesture input / sketch recognition in graphical user interfaces [Buxton86a] Buxton, William, and Myers, Brad A. "A Study in Two-Handed Input", to appear in 1986 Conference on Computers and Human Interaction, CHI'86, Boston April 12-17, 1986 Digitizer tablet plus other controls, for selection of text: multi-touch/multi-hand. [Buxton86b] Buxton, William "There's More to Interaction than Meets the Eye", in User Centered System Design: New Perspectives on Human-Computer Interaction, Norman, Donald A. and Draper, Stephen W, Eds, Lawrence Erlbaum Associates, Hillsdale NJ and London, 1986 Comparison/study of locator/pointing devices and user interfaces: isometric, and acceleration joysticks (and multiple isometric joysticks: different size and force), trackballs, nulling problem for multiple sliders. Tablet best, in part because it can emulate all other devices, as general purpose tools can be used for more limited applications. Notes that for two-hand (multi-touch) inputs, input grammars don't do well. Semantically equivalent devices: e.g. etch-a-sketch (two knobs) and a different toy using a single joystick. Semantically identical/equivalent, but usability and use much different. [BuxtonWB86c] Buxton, William "A Study in Two-Handed Input", University of Toronto, 1986: available at billbuxton.com Video presentation showing user interface using two hands: mixer panel for left hand, tablet with puck for right hand. Companion video to "A study in Two Handed input". [BuxtonWB86d] Buxton, William "Buxton/Ward personal correspondence", Jean Renard Ward, 1986 Personal correspondence with Bill Buxton, 1986/1987. Conference panel, two-handed (multitouch) tablet, etc. [CIC86a] Communications Intelligence Corporation "Product literature CIC Handwriter for IBM 5550/5540 terminal, IBM JX terminal, Handwriter for Japanese PC", Communications Intelligence Corporation, 1986 (in Japanese) Line of handwriting recognition tablets using coprocessor board for Japanese Katakana/Hiragana handprinted characters. Compare with same company Proficient and other applications in U.S. [CIC86b] Communications Intelligence Corporation "Correspondence and press file, Communications Intelligence Corporation", Communications Intelligence Corporation, Correspondence file with CIC entry into US market after Japan. Technical information on Handwriter 8:26:26 PM]

126 ( ) History of Pen and Gesture Computing handwriting recognition forms data entry combining data collection (paper forms) and data entry (PC), "legibility exercises" to train users on ambiguous character e.g. 5-S, WRITE-ON keyboard replacement users have difficulty learning a keyboard (circa 1986). application architecture includes signature verification, CAD input (see also Proficient), forms data entry, templates for Lotus/123 spreadsheet. Handwriting recognition board supports third-party tablets. [Carpenter86] Carpenter, G.A. and Grossberg, S. "A Massively Parallel Architecture for a Self-organizing Neural Pattern Recognition Machine", manuscript in preparation, Authors' address: Center for Adaptive Systems, Department of Mathematics, Boston University, Boston, Massachusetts Adaptive recognition for static OCR recognition: gives example of "simple" 5x5 characters, but patterns. Comments: Does not match human recognition of patterns (random 5x5 bits as input). [Casey86] Casey, R.G. "Text OCR by Solving a Cryptogram", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Adaptive recognition for static OCR using clear-text and "unknown" type-font text for a cryptogram: but is clear-text always known? [Casio86a] Casio "Casio IF-8000 Calculator", Collection from Pocket Computing Museum, Calculator (this reference says 1986) with touchscreen input, and also zone-based handwriting recognition on a touch-film keyboard/keypad Same reference includes Apple Newton Message Pad 110 [Casio86b] Casio "Digital Diary Casio IF-8000 Operation Manual", Casio, 1984/1986 Two-side (Refalo-style) calculator with small touchscreen display and calculator keypad on the left, alphabetic keypad on the right. Electronic ink note-taking on the touchscreen. Memos and notes can include both (typed) text and electronic ink notes. Calendar and address book. Compare with similar Casio calculator with handwriting recognition for calculator input. [Chunbiao86] Chunbiao, G. and Guorong, X. "Automatic Recognition of Printed Chinese Characters by Four Corner Codes", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Kanji/Chinese characters in minimal Chinese standard (3000 common): much more than Japanese standard (1493) Chain codes/segments: Chinese/Kanji characters in only nine kinds of primitive segments [CinaVJ86a] Cina, Vincent J. Jr.; White, Steve R.; and Comerford, Liam "ABYSS: A Basic Yorktown Security System: PC Software Asset Protection Concepts", IBM Research Report Number RC 12401, December 18, 1986 ABYSS: A Basic Yorktown Security System for PC software asset protection: copy-protection, DRM. Cites to cryptography, does not give details. Protected (trusted) processor, execution right, application data store. Relies on electronic tokens (e.g. smart cards) [CohenM86] Cohen, M. and Grossberg, S. "Neural dynamics of speech and language coding: developmental programs, perceptual grouping, and competition for short-term memory", Human Neurobiology, 1986, Vol 5, pp 1-22 According to Daniel Bullock's letter from BU in 1986, Cohen did handwriting recognition in Theoretical adaptive work: "random growth" of human brain [Cordella86] Cordella, L.P. and Marcelli, A. "Normalization and Decomposition of Thin Lines Representing Handprinted Characters", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Decomposing characters into segments (curves, arcs) that are invariant with respect to handwriting variability (compare to chain-code segments) Structural analysis beats the pants off of statistical / adaptive methods Cites Suen/Shillman paper on U-V discrimination Recognition by grammar/syntax of (chain-code) segments [Crane86] Crane, H.D., and Ostrem, J.S. "Confusion Grouping of Strokes in Pattern Recognition Method and System", United States Patent 4,573,196, February 25, 1986, assigned to Communication Intelligence Corporation, Menlo Park, California. Confusion matrix for Chinese/Kanji character recognition, using distance and curvature metrics [Currit86] Currit, P.A., Dyer, M., and Mills, H.D. "Certifying the Reliability of Software", IEEE Trans. Software Engineering, Vol SE-12 No 1, January 1986, pp 3-14 The accepted approach to software development is to specify and design a product -- and then to test selectively with cases perceived to be typical of those requirements. Frequently the result is a product which works well against inputs to be tested but which is unreliable in unexpected circumstances Testability of software: software-engineering regressions tests Statistical testing of software can predict reliability: however, it cannot tell you about the existence, severity, or number of remaining errors Sonic tablet digitizer: 0.1mm accuracy over 1.0 meter square active area (but then says +-2.0mm after all calibration!) Sonic tablet digitizer: local drafts throw off same as 1 degree C temperature Sonic tablet digitizer: calibrate to correct for air speed vs temperature [DeBruyne86] de Bruyne, P. "Compact Large-Area Graphic Digitizer for Personal Computers", IEEE Computer 8:26:26 PM]

127 ( ) History of Pen and Gesture Computing Graphics and Applications, Vol 6 No 12, December, 1986, pp Sonic digitizer tablet: ultrasonic shock wave: wireless/cordless cursor/stylus, using spark gap sonic pen, accuracy/error is 0.1% of distance measured [Derouault86] Derouault, A. and Merialdo, B. "Natural Language Modeling for Phoneme-to-Text Transcription", IEEE Trans. Pattern Analysis and Machine Intelligence", Vol 8 No 6, November 1986, pp Statistical modelling and grammatical rules combined for context in French natural language [DhawanSK86a] Dhawan, Satish K. "Field-coupled pointing device", United States Patent 4,564,835, January 14, 1986 Mouse tablet: tablet has array of electromagnetic (coil) or electrostatic single elements in a grid matrix, mouse motion determined by number of changes of phase passed over. Mentions some elements being driving out of phase with others to resolve "ambiguity points" when mouse sensor is directly over a physical pixel/element. [DosterW86a] Doster, W. "Designing a Document Analysis System", tutorial at 8th International Conference on Pattern Recognition, Paris, France, October, 1986 Cited in Baird86a [DosterW86b] Doster, W. "private letter of May, 1986", AEG Aktiengesellschaft, Forschungsinstitut Ulm, Sedanstrasse 10, D-7900 Ulm, West Germany. Doster also has a DCR (instead of OCR) character data base collection. [DowningD86a] Downing, Douglas and Covington, Michael "Dictionary of Computer Terms", Barron's Publishing 1986 (hardcopy book) Technical dictionary of computer terms 1986 [Ewing86] Ewing, J., Mehrabanzad, S., Sheck, S., Ostroff, D. and Shneiderman, B. "An Experimental Comparison of a Mouse and Arrow-Jump Keys for an Interactive Encyclopedia (abstract only)", International Journal of Man- Machine Studies, Jan 1986, pp abstract only: see also Ewing85 User-interface on pointing methods: applies to mouse emulation on a tablet? Poor typists prefer mouse for pointing, good typists prefer arrow keys. [Firdman86] Firdman, H.E. "Components of AI Systems", AI Expert, Vol 1 No 1, 1986, pp General comments on what an AI system "is" DCR as Artificial intelligence: "domain specific: rules of thumb extracted from experts" [FlurryGA86a] Flurry, Gregory A. "Real-time rub-out erase for an electronic handwriting facility", United States Patent 4,633,436 assigned to International Business Machines Corp, Armonk, New York, December 30, 1986 Stylus patent for tablet digitizer: Looks very similar to erase function of Wang Freestyle: real-time rubout erase on back end of stylus. [FoleyJD86] Foley, J. et al "Managing the Design of User-Computer Interfaces", Report available from Computer Graphics Consultants, Incorporated, 616 G Street SW, Washington DC 20024, 1986 Jim Foley's consulting report on user interface subsystems [Forethought86a] Forethought Inc. "FileMaker Plus Guide to FileMaker Plus Templates", Forethought Inc, 1986 Database development system for MacIntosh by Nashoba Systems. [Fu86] Fu, K.S. "A Step Towards Unification of Syntactic and Statistical Pattern Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol 8 No 3, May 1986, pp Previously published in Vol 5 No 2, March, 1983 Could also be listed as Fu83: was re-printed in same journal Attributed grammar, control diagram, semantics, statistical pattern recognition, syntactic pattern recognition, syntax-semantics tradeoff Subpatterns, substructure (primitives) of pattern as features There is yet no systematic method to select appropriate syntactic and semantic complexities (features) for a specific pattern recognition problem Gives examples of distorted hand-written "E"s, and a possible tree/hierarchical representation of the structure for recognition [GanesanA86a] Ganesan, Apparajan and Weise, Wayne A. "Method and apparatus for capacitive keyboard scanning", United States Patent 4,651,133, March 17, 1987 Capacitive keypad/keyboard of keys on a grid: ratiometric measurement of total relative capacitance of all other keys in the row, thereby eliminating problems from variations and capacitance. Also detection of multiple key closures (without limit) by comparing both by row and by column to determining a key closure. Capacitors can be cross-point connected: multi-touch [GarciaJD86a] Garcia, John D. "Personal Identification Apparatus", United States Patent 4,621,334, November 4, 1986 Authenticate user biometrically using keystroke latency/delay pattern of typing on keyboard. Cites to Saltzer [GarrettLN86a] Garrett, L. Nancy, Smith, Karen E. and Meyrowitz, Norman "Intermedia: Issues, Strategies, and Tactics in the Design of a Hypermedia Document System", Proc. of 1986 ACM Conf. on Computer-supported cooperative Work, pp :26:26 PM]

128 ( ) History of Pen and Gesture Computing Hyperlinks in electronic document images in the text [GillEN86] Gill, E.N. "An Evaluation of the Documented Requirements of the SSP UIL and a Review of Commercial Software Packages for the Development and Testing of UIL Prototypes", NASA/ASEE Report N , Contract NST , August 8, 1986 Summary report of 4GL / UIL User Interface Languages for the proposed Manned Space Station: cites Pencept PenPad for touch/stylus based user interface using a digitizer tablet, including Word and PenCAD gesture-based ("Recognition Macro") interfaces [GrooverMP86a] Groover, Mikell P.; Weiss, Mitchell; Nagel, Roger N.; and Odrey, Nicholas G. "Industrial Robotics: Technology, Programming and Applications", McGraw-Hill, 1986 (hardcopy book) Cited in Lindemann10. multitouch / tactile sensor: page 152, figure 6-5, Lord Corporation LTS 300 Series with 80x80 resolution, each point reports force magnitude. [Guanxiong86] Guanxiong, Z. and Hongyuan, W. "Angle Coding and it's Application in Pattern Recognition", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Chain-codes: eight-direction angle codes (Freeman codes) [Hase86] Hase, M., Suzuki, G., and Hisayasu, I. "A Method for Extracting Marked Regions from Document Images", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Separating graphics and text characters from an image [HeckbertPS86a] Heckbert, Paul S. "Survey of Texture Mapping", Proc. Graphics Interface '86, Vision Interface '86, May 1986 pp Survey of papers on texture mapping: texture mapping two phases: map texture space to object space (surface), then map object space to 2D screen space. Transformations can be combined into one. Similar to image warping, correction for geometric distortion. Orthographic/Perspective projections. [HeckbertPS86b] Heckbert, Paul S. "Survey of Texture Mapping", IEEE CGA, Nov 1986, pp Later version of Graphics Interface paper, same title. [HekmatpourS86a] Hekmatpour, S. and Ince, D.C. "Forms as a Language Facility", A.C.M. SIGPLAN Notices, Vol 21 No 9, September 1986, pp Quotes Gehani82: Proposes forms (for data entry: fields etc.) as so important to deserve own data type / objects in a language. (Object-based programming answers this.) Compare explanation with Pencept and other forms packages for handwriting recognitions. [Hidai86a] Hidai, Y., Ooi, K., and Nakamura, Y. "Stroke Re-ordering Algorithm for On-line Hand-written Character Recognition", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp BLRT/chain-code: segments at eight discrete slants [HidaiY86b] Hidai, Y., Ooi, K., Nakamura, Y., and Kurosawa, Y., "Hand-written characters segmenting system", European Patent , April 9, 1986, assigned to Toshiba. Hand-written characters segmentation system has tablet determining boundary between characters according to stroke duration (inter-stroke timeout) and coordinates data (proximity height from tablet). [HillRD86a] Hill, Ralph D. "Supporting Concurrency, Communication, and Synchronization in Human-Computer Interaction: The Sassafras UIMS", ACM Trans. on Graphics Vol 5 No 3, July 1986, pp UIMS User Interface management system supporting concurrent (multi-stream) input from multiple devices: e.g. mouse/digitizer and keyboard. Example of two-handed (multi-touch) paint program. Multiple concurrent virtual devices on a touch tablet (see Buxton). [Holbaek-Hanssen86] Holbaek-Hanseen, Erik, Braten, Knut, and Taxt, Torfinn "A General Software System for Supervised Statistical Classification of Symbols", Proceedings of ICPR International Conference on Pattern Recognition, Paris, October 27-31, 1986, pp Handwriting character recognition using grid lengths of chain code skeletons, boundary tracing, "auxiliary features" [Honeywell86a] Honeywell Information Systems "MULTICS Administration Maintenance and Operations Commands", Honeywell Information Systems GB64-00A, November 9186 Command-line documentation for Multics segmented operating system: all files memory-mapped, access control lists (for files and all code segments) use syntax expressions for access permissions and privileges. access granted to person.project.tag, supports wildcards. Both discretionary (ACL) and nondiscretionary/mandatory (AIM access isolation mechanism) access control. AIM has minimum/maximum range level. [HongoY86a] Hongo, Yasuo and Nitta, Yoshio "Pattern Recognition Apparatus", United States Patent 4,628,533, December 9, 1986, assigned to Fuji Electric Company, Limited, Kanagawa, Japan. Character recognition by pixel matching on OCR [HudsonSC86a] Hudson, Scott E. and King, Roger "A generator of direct manipulation office systems", ACM Trans. on Information Systems, Vol 4 No 2, April 1986, pp :26:26 PM]

129 ( ) History of Pen and Gesture Computing HIGGENS - Human Interface Graphical Generation System: Direct manipulation (generator) UI for office entities, included undo/reversal/recovery mechanism. Drag object to trash can to delete. Xerox Star as direct manipulation system. [HullJJ86a] Hull, Jonathan J. "Hypothesis generation in a computational model for visual word recognition", IEEE Expert, 1986, pp Context in optical character recognition of type fonts: Cited in Sinha88 [HullJJ86b] Hull, Jonathan J. "The Use of Global Context in Text Recognition", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Context via "nearest neighbor" words from a dictionary Cognitive: OCR recognize words by overall outline of vertical bars, etc., not actual individual letters Cites Shillman on use of human cognitive psychology, but this time it's on word-level recognition [HullJJ86c] Hull, Jonathan J. and Srihari, S.N. "A computational approach to word shape recognition: Hypothesis generation and testing", Proceedings of IEEE-CS Conference on Computer Vision Pattern Recognition, June 1983, pp Hypothesis testing for optical character recognition of font characters: tree search -- compare with PenCept pair-wise binary comparison? Cited in Bozinovic89 [HullsLR86a] Hulls, L.R. "Polyphase Digitizer", United States Patent 4,570,033, assigned to Numonics Corporation, Lansdale Pennsylvania, February 11, 1986 Phase-shifting tablet digitizer, using three-terminal grid winding. Correction for edge effects (hockey-stick) by adjusting X value when Y value indicates close to edge, and vice versa. [Hutchins86] Hutchins, E., Hollan, J. and Norman, D.A. "Direct Manipulation Interfaces", in User-Centered System Design, D.A. Norman and S. Draper, editors, Erlbaum Associates, Hillsdale, New Jersey, 1986, pp Cited in JohnsonJ89 Direct-manipulation graphical user interface: model objects must behave as user expects real objects to behave that the image represents: automating response is not enough. Cites to Buxton and Minsky on using pressure and torque for touch input. [ICSA86] ICSA "CRS (Character Recognition System) software product description", Intelligent Computer Systems and Applications, Inc., Box 1555, Station H, Montreal Quebec H3G 2N5 Canada, 1986 Ching Suen's commercial company to sell OCR character recognition software. [Inagaki86] Inagaki, Naoki "Key-In Device", United States Patent 4,578,811, March 25, 1986 Zone-based / matrix based character recognition, using the keypad of the Casio 8000 calculator as the input table. Compare: Unistroke/ single stroke character recognition recognition software. [JacobRJK86a] Jacrob, Robert J.K. "A specification language for direct-manipulation interfaces", ACM Trans. on Graphics, Vol 5 No 4, October 1986, pp Direct-manipulation user interfaces GUI definition, using inheritance of graphical objects. Describes context-sensitive gestures (p. 308). [JaegerRP86a] Jaeger, Robert P. "Patterned Conductive Ink Touch Panel", United States Patent 4,625,075, November 25, 1986 Resistive sheet/film touch digitizer using transparent conductive ink. Pattern of electrodes at edges compensates for distortion of resistance. [JandrellLHM86a] Jandrell, L.H.M. "Data Input Device with a Circuit Responsive to Stylus Up/Down Position", United States Patent 4,575,580, assigned to Astec International, Limited, Kowloon, Hong Kong. Resistive sheet tablet using a force/pressure-sensitive conductive sheet Gives fix for wild-points at starts and ends of strokes from up/down contact resistance on resistive sheet tablets [JelinekF86a] Jelinek, F. "Markov Source Modeling of Text Generation", unpublished manuscript, December Author's address: Continuous Speech Recognition Group, IBM T.J. Watson Research Center, Yorktown Heights, NY Speech: probabilistic language models for vocabulary context Cites need for billions of samples to get statistically reasonable vocabulary probabilities [JelinekF86b] Jelinek, F. "Self-Organized Language Modeling for Speech Recognition", unpublished manuscript, December Author's address: Continuous Speech Recognition Group, IBM T.J. Watson Research Center, Yorktown Heights, NY Speech recognition: has large data-bases of business correspondence for vocabulary [Jorgensen86] Jorgensen, C. and Matheus, C. "Catching Knowledge in Neural Nets", AI Expert, December 1986, pp General overview of neural networks, perceptrons, adelines, etc. "credit assignment problem" in perceptrons and adaptive recognition Neural nets making a comeback: future still not proven [Kable86] Kable, R.G. "Electrographic Apparatus", United States Patent 4,600,807, assigned to Scriptel Corporation, Columbus, Ohio, July 15, 1986 Scriptel patent on resistive-sheet, electrostatic/capacitive-coupling digitizer tablet. Mentions correction matrix for entire surface of a digitizer tablet 8:26:26 PM]

130 ( ) History of Pen and Gesture Computing [KaehlerT86] Kaehler, Ted and Patterson, Dave "A Taste of Smalltalk", Norton and Company, New York, 1986 Introduction to Smalltalk language and environment using Towers of Hanoi programming example. (Electronic file.) [KankaanpaaA86] Kankaanpaa, Arto "An Advanced display System with Natural Interactivity", Proc. of Eurographics 86,European Computer Graphics Conference and Exhibition, Lisbon, August, 1986, pp Text editing with single-stroke (mostly) gestures on integrated touchscreen digitizer and LCD display, suggests electroluminescent or plasma display. [Kao86] Kao, H.S.R., Van Galen, G.P., and Hoosain, R. "Graphonomics: Contemporary Research in Handwriting", North-Holland Publishers, Amsterdam-New York-Oxford-Toronto, 1986, Number 37 of "Advances in Psychology", Stelmach, G.E. and Vroon, P.A., editors. [Kasturi86] Kasturi, R., Shih, C., and Fletcher, L.A. "An Approach for Automatic Recognition of Graphics", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Separating graphics and text characters from an image [KatayamaH86a] Katayama, Hildeji; Katoh, Keisuke; Kita, Junya; and Shimada, Motowo "Digital contrast control circuit for display unit", European Patent Application EP A2, Feb 26, 1986 Contrast control from keyboard using up/down counter circuit. [Kay86a] Kay, Alan "A Talk by Alan Kay: summary of video interview Alan Kay talks about Sketchpad, Grail, the Dynabook", The History of the Personal Workstation, 27 May 1986 Grail system with freehand input of letterforms, boxes, lines, corrections to previous drawings, Sketchpad, Dynabook. Video includes Grail system showing handwriting recognition (with lightpen) and scratchout gesture. Screenshots from video in file. File also has notes on KiddiKomp (later version of Dynabook), Flex Machine (with tablet) ca [Kay86b] Kay, Alan "Alan Kay explains Sketchbook, Grail, and the Dynabook", Available from YouTube (2009): Grail system with freehand input of letterforms, boxes, lines, corrections to previous drawings, Sketchpad, Dynabook. Video includes Grail system showing handwriting recognition (with lightpen) and scratchout/rub-out gesture. [Kondo86] Kondo, S. and Attachoo, B. "Model of Handwriting Process and its Analysis", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Features: distortions of end points on strokes do not affect human classification of handwriting, of middles of strokes does affect recognition Stroke structure (order) very important in handwriting process, and therefore should be important in character recognition Big problem in handwriting recognition is finding/extracting features not subject to handwriting distortion. Since set of all characters is not clear cut, any definition based on "invariant" features of a class is likely to be contradictory. Simulate handwritten characters: problem is finding out what base forms are that you should vary on. [KovedL86a] Koved, Larry and Shneiderman, Ben "Embedded Menus: Selecting Items in Context", CACM, Vol 29 No 4, April 1986, pp Touchtext: Pop-up menus showing pick-list of alternative words in a spellchecker UI. [Kurosawa86] Kurosawa, Y. and Asada, H. "Attributed String Matching with Statistical Constraints for Character Recognition", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Chain code: use primitive segments and to string matching Exhaustive depth-first search of syntactic string for primitive segments Katakana and alphanumeric characters with BLRT-like primitive segments Combinatorial explosion of production rules Combination of primitive-segment syntactic matching and statistical constraints BLRT-chain codes give multiple-matches (3 or so), so second, ad-hoc comparison analysis for each possible pair of characters [LamSW86a] Lam, S.W. and Baird, H.S. "Performance Testing of Mixed-font, Variable-size Character Recognizers", Computing Science Technical Report No. 126, AT&T Bell Laboratories, Murray Hill, New Jersey, November 6, 1986 Gives statistics for size of test set to justify recognition performance claims of a given percent with a justifiable confidence level To claim 99.5% recognition accuracy with 95% confidence, need 1500 samples To claim recognition accuracy improved from 99% to 99.5% with 95% confidence, need 5000 samples Refers to clustering and partitioning algorithms for feature selection for Bayesian statistics OCR static recognition system was tested without ligatures, without several "special" characters 99.5%+ performance requires very large, carefully-designed tests Test performance of 99.5% was with common cases omitted, and some confusions forgiven For OCR, scanner digitizer performance is critical OCR scanner/digitizer: has short term fluctuations in sensitivity (re-scanning? was there internal averaging?) Testing/training done only with high-quality data, not low-quality For OCR digitizer scanners, scanning performance is not 8:26:26 PM]

131 ( ) History of Pen and Gesture Computing uniform or repeatable page-to-page OCR tested and trained only on data of good quality, but question was raised Resolution: 24 pixels not enough data accuracy for OCR OCR at 200 dots/inch resolution started to fail at 12 point type size Refers to tools to collect automatically a very large (168,000) character test set data base [LaubeM86a] Laube, M. "Audiographic Terminal", ITT Electrical Communication, Vol 60 No 1, 1986, pp Says maximum frequency of handwriting is 15 Hz, therefore 40 points/second digitizing rate (don't believe it!). Resistive-sheet tablet for handwriting, force/pressure-sensitive tablet (!!). Handwriting/voice/display terminal, like Interspec. [Leedham86] Leedham, C.G. and Downton, A.C. "On-line recognition of Pitman's handwritten shorthand -- an evaluation of potential", International Journal of Man-Machine Studies, Vol 24, 1986, pp degree variation in angle of nominal vertical and horizontal strokes Short and long strokes overlap in length Textbook writing (shorthand) is NOT the most recognizable to humans Human recognition: textbook shorthand is 93.5%, student is 97.5%, expert is 48% For small writing, hooks and loops are as big as many of the strokes Unconstrained shorthand input Intended straight strokes consistently tend to be curved Has gall to say "recognition would be better if writing were neater" Ignored strokes if too close to prior/preceding stroke Stroke length variability is 20% in length Says strokes were "lost" (thrown out) if not classified as "legal" Did not deal with overdrawn/intentionally retraced strokes Says recognition would get better with user practice (human factors/user interface) "no amount of post-processing (context?) will fix segmentation and stroke classification errors" Test protocol intentionally slow to get better quality data Test protocol defeated linguistic context, resulting in users having to concentrate more Refers to need for better (non-simplified) recognition algorithms Cites ergonomic problems of tablet styli Cites "writer's unfamiliarity with the system and capabilities" as performance factor [LetteraC86a] Lettera, C., Masera, L., Paoli, C. and Proinelli, R. "Use of a Dictionary in Conjunction with a Handwritten Text Recognizer", Proc. ICPR(86), pp Cited by Marlin Eller, Microsoft Pen Computing group [LiebermanH86] Lieberman, Henry "Using Prototypical Objects to Implement Shared Behavior in Object Oriented Systems", ACM SIGPLAN Notices, Vol 21 No 11, November 1986, pp Prototype-based languages versus classes: examples include turtle and pen object for drawing, cites to SELF and Smalltalk [Linus86a] Linus Technology "Personal correspondence Linus technology", Linus Technology, 1986 Correspondence file on Linus Technology: handwriting recognition / touchscreen laptop. [Litvin86] Litvin, Y. "Private letter of August 4, 1986", Author's address: Skylight Software, 2 Charles Street, Bedford, Massachusetts DCR/on-line handwriting recognition not deterministic: there is no formal criteria for correctness (not true: he misunderstood "deterministic") [LongleyD86a] Longley, Dennis and Shain, Michael "Dictionary of Information Technology, Second Edition", Oxford University Press, 1986 (hardcopy book) Technical dictionary [LuH86] Lu, H.E. and Wang, P.S.P. "A Comment on 'A Fast Parallel Algorithm for Thinning Digital Patterns'", CACM, Vol 29 No 3, pp Skeletonization/line-thinning, special notes on endpoints NOT to be deleted: compare with smoothing/preprocessing algorithms for digitizer tablet data by Pencept [MaYL86a] Ma, Y.L., Jang, S.Y., and Ma, C. "Pattern Recognition by Circular Layer Code Approach", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Chinese OCR, 97.5% on 4 characters tested with 20 samples each (!!!) [Maamari86] Maamari, F. and Plamondon, R. "Extraction of the Analog Pentip Position, Velocity and Acceleration Signals from a Digitizer", Neuronal and Motor Aspects of Handwriting, North-Holland Publishers, J.S.R. Kau, Editor, 1985 Analog filter on digitizer output gives "improved accuracy" (!!??!!) Low-pass filtering on tablet digitizer lets you get "better" time resolution (?) [MacNelly86] MacNelly, Jeff ""Shoe" comic: "I wish I sat behind someone with decent handwriting"", "Shoe", November 21, Taken from Boston Globe newspaper, Boston, USA, Page 102 Student in a school class during a test: Fill in the blanks: "To the victor belong the *snails*." That can't be right... I wish I sat behind someone with decent handwriting. (Humorous illustration of the problems of imperfect handwriting recognition, and of the difficulty of human recognition of handwriting) [MaedaY86] Maeda, Y., Yoda, F., Matsuura, K., and Nambu, H. "Character Segmentation in Japanese Handwritten Document Images", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp [Maier86] Maier, M. "Separating Characters in Scripted Documents", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp :26:26 PM]

132 ( ) History of Pen and Gesture Computing AI artificial intelligence was seen as panacea Cited in Wolf86 [Mandler86] Mandler, E. "conversation of September, 1986", AEG Aktiengesellschaft, Forschungsinstitut Ulm, Sedanstrasse 10, D-7900 Ulm, West Germany. One-stroke "T" more common in Europe [Mangione86b] Mangione, P.A. "SSI's Phonetic Engine (TM)", Speech Technology Magazine, Mar/April 1986 Speech Systems phonetic feature speech recognition [Mangione86c] Mangione, P.A. "Phonetic Recognition - The Basic Building Block", Speech Technology Magazine, April/May 1986 Speech Systems phonetic feature speech recognition [Mangione86d] Mangione, P.A. "What about the user?", Proceedings of Speech Tech '86, April , New York, New York, pp Marketing V.P. did paper on Speech Systems' phonetic speech input product [MantasJ86a] Mantas, J. "An overview of character recognition methodologies", Pattern Recognition, Vol 19, 1986, pp Cited in Baptista88 [MantasJ86b] Mantas, J. "A Fuzzy Decision Operator in Recognizing Greek Hand-drawn Characters", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Pattern recognition is not just pattern classification, but also design of recognition systems Statistical versus syntactic/linguistic pattern recognition (does not like statistical) Statistical/Bayesian recognition cannot be done for handwriting because a priori probabilities cannot be calculated for every feature [Meisel86] Meisel, W.S. "Implications of Large Vocabulary Recognition", Proceedings of Speech Tech '86, April 28-30, 1986, New York, New York. Speech Systems speech recognition using phonetics (sloppy cursive): co-articulated speech, vs isolated, connected, continuous Speech: storing multiple-word (multiple-char) segments as special "words" (BLRTs/variability): phoneme detection reduces variability to a set of rules at phoneme level Ambiguity: groups of words (chars) vs single words: "there are four" vs "therefore" Dictionary look-up/search: speed and accuracy of recognition unrelated to Size of dictionary, but only to complexity of syntax Adaptive vs user-independent: controversy over "speaker-independence" Adaptive: must begin at a usable level even so, otherwise bad human-factors (frustration) Adaptive: correction for content, not for error (user wants a different word) Adaptive: separate enrollment session a bad user-interface idea for "normal" speakers Penalty for speaker-independence too high (frustration) Variability: phoneme speech system can extrapolate/predict words after adaptive samples Trade-off between accuracy and constraints on user in speech: What if user uses bad English? [Microsoft86a] Microsoft Corporation "Microsoft Excel User's Guide Version 1.03 with Business Graphics and Database for the Apple Macintosh", Microsoft Corporation, 1986 (hardcopy manual) Excel 1.03 spreadsheet processing for the Apple Macintosh - shows integration with Apple menus. Contains update for version 1.5 describing File contains additional materials: [MooreCA86a] Moore, Cecil A. "Two-layer Touch Tablet", United States Patent 4,587,387, May 6, 1986 Two-layer resistive grid digitizer tablet. Carbon ink used as conductor, is resistive 100K-200K per bar/conductor, or conductive silver ink, voltage divider along conductors. [Morasso86] Morasso, P. "Understanding Cursive Script as a Trajectory Formation Paradigm", in "Graphonomics: Contemporary Research in Handwriting", Elsevier Science Publishers B.V. (North-Holland), 1986, pp Hand gestures and handwriting: study of movements. Measures curvature profile, not just velocity profile, of handwriting motion. Claims no handwriting information at frequency above 10 Hz, shows peak at 5 Hz and D.C. (but his samples are mostly loop motion, not fast block printing). Shows plots of low-pass frequency filtering on handwriting data at 10 Hz and below (frequency content of dynamic writing motion: implies only low-rate digitizer tablet needed if it is accurate enough?) Author intends to collect lots of allograph handwriting samples to build a future recognizer for cursive script. [Morita86] Morita, Toshiaki; Horii, Masahiro; Tasaka, Shigeru; Hirose, Hitoshi "Handwriting Character Recognition Device", United States Patent 4,607,386 assigned to Sharp Kabushiki Kaisha, Osaka, Japan, August 19, 1986 Chinese handwritten character recognition without regard to stroke order [Murase86] Murase, H., and Wakahara, T. "Online Hand-Sketched Figure Recognition", Pattern Recognition, Vol 19 No 2, pp , 1986 Sketch context: must be valid connections to other flowchart symbols 97.2% accuracy: test protocol was only 20 people, nine symbols, four times each Mentions lots of user-interface work on on-line sketch recognition For "standardized" sketch symbols, high variability (no regularity) in stroke order Says stroke order variable, but end/connection points are always the same Refers to stroke connection variability for "sketched" symbols [MyersBA86] Myers, Brad A. and Buxton, William "Creating Highly-Interactive and Graphical User Interfaces by Demonstration", Proceedings of SIGGRAPH '86, pp 249 ff. 8:26:26 PM]

133 ( ) History of Pen and Gesture Computing Direct-manipulation user interface. Simulated mouse device with three buttons on display screen. [NagyG86a] Nagy, G., Seth, S., and Stoddard, S.D. "Document Analysis with an Expert System", in "Pattern Recognition in Practice", Gelsema, E.S. and Kanal, L.N., editors, Elsevier Science Publishers B.V. (North-Holland), 1986 Tiling/windows applied for format of written/printed documents [NagyG86b] Nagy, G.R., Seth, S., Einspahr, K., and Meyer, T. "Efficient Algorithms to Decode Substitution Ciphers with Application to OCR", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Adaptive recognition using clear-text and "unknown" type-font text for a cryptogram: but is clear-text always known? [Nakagawa86a] Nakagawa, Masaki, Manabe, Toshihiko, Aoki, Katsuo, Ikeda, Yuji, and Takahashi, Nobumasa "Online Handwritten Character Recognition as a Japanese Input Method", Proceedings of???, pp on-line handwriting recognition system for Japanese, JOLIS-1: chain codes (8-directional), 29 primitive strokes types by dictionary look-up [Nakagawa86b] Nakagawa, M., Aizawa, T., Komoda, C., Ideda, Y., and Takahashi, N. "Syntactic Pattern Recognition with Stochastic Dissimilarity in Japanese On-line Input Systems (JOLIS)-1/1.5", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Chain-code: Japanese recognition by classifying strokes into primitive strokes Chain-code: eight stroke directions/segments Chain-codes: extension to elastic matching [NespoulousJ86a] Nespoulous, J.; Perron, P.; Lecours, A.R. "The Biological Foundations of Gestures: Motor and Semiotic Aspects", Lawrence Erlbaum Associates, Hillsdale, NJ 1986 Defines gesture as "the notion of gesture is to embrace all kinds of instances where an individual engages in movements whose communicative intent is paramount, manifest, and openly acknowledged": Cited in DeNardi master's thesis p. 11 [Nestor86] Nestor, Inc. "Nestor Press clipping file", Nestor, Inc. Synthetic neuron pattern-recognition technology, applied to OCR and DCR handwriting recognition [OedR86a] Oed, R. "telephone conversation of July 17, 1986", AEG Aktiengesellschaft, Forschungsinstitut Ulm, Sedanstrasse 10, D-7900 Ulm, West Germany. You get different writing forms after the first 3-4 lines out of 20 [OedR86b] Oed, R. "conversation of September, 1986", AEG Aktiengesellschaft, Forschungsinstitut Ulm, Sedanstrasse 10, D-7900 Ulm, West Germany. Gesture/symbols are modified characters: slash-m, "DF" [Ogozalek86] Ogozalek, V.Z. and Van Praag, J. "Comparison of elderly and younger users on keyboard and voice input computer-based composition tasks", in Human Factors in Computing Systems, Proceedings of CHI 83 Conference, 1983, Boston. Cited in Martin89 Martin89 cites as saying speech recognition and typing just as fast a user-interface for typing documents [OlsonB86] Olson, Bruce T. "Is your Handwriting Legible Enough for the Computer?", Journal of Law enforcement Report writing, Vol 3, No 1, December 1986 Pencept PenPad for capturing police reports using handwriting recognition [OritaM86a] Orita, M., Kanasaki, M., Toda, Y., Mishima, T., Suzuki, M., Onuma, C., and Takatoo, M. "Image extraction using density distribution characteristics", European Patent , October 22, 1986, assigned to Hitachi, Limited, Tokyo, Japan. OCR scanner preprocessing to extract character images from background clutter [PCMagazine86] PC Magazine "Best of Pencept Penpad", PC Magazine, January 14, 1986, Ziff-Davis Publishing Company Review of best PC products of describes Pencept PenPad with PenCad software, use of "pen macros" / letter gesture commands [PavlidisT86a] Pavlidis, Theo "A vectorizer and feature extractor for document recognition", Computer Vision, Graphics, Image Processing, Vol 35, 1986 Cited in Kahan87 [PavlidisT86b] Pavlidis, Theo "A Critical Survey of Image Analysis Methods", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Image processing / recognition design requires many methods, not just one theoretical basis A digitizer tablet good enough for writing input [PearsonG86a] Pearson, Glenn and Weiser, Mark "Of Moles and Men: The Design of Foot Controls for Workstations", Proc. CHI '86, April 1986, pp Various foot controls (foot mouse, "tush-mouse", foot touchscreen, etc.). Mentions multipoint/multi-touch touchpad, using multiple-finger gestures for typing regardless of location (Shackelford) [Pencept86a] Pencept "PENWARE (tm) Pendraw (tm) II User's Guide", Pencept Division of Numonics, 39 Green 8:26:26 PM]

134 ( ) History of Pen and Gesture Computing Street, Waltham, Massachusetts 02154, 1986 [Pencept86b] Pencept "Penpad 300 User's Guide", Pencept Division of Numonics, 39 Green Street, Waltham, Massachusetts 02154, 1986 [Pencept86c] Pencept "Penpad 320 User's Guide / User's Guide for the Pencpad 320 for IBM PC and compatibles", Pencept Division of Numonics, 39 Green Street, Waltham, Massachusetts 02154, 1986 Handwriting recognition tablet: shows single-stroke and multiple-stroke characters shapes in one set. Includes system disk with mouse driver to adapt tablet input (including command templates, character recognition) to mouse/keyboard input. [Pencept86d] Pencept "Writing Samples", Pencept Division of Numonics, 39 Green Street, Waltham, Massachusetts 02154, 1986 Collections of unconstrained, natural writing samples taken from hotel receipts, university bulletin board notes, bills of sale, and cross-word puzzles from the magazines in waiting rooms and airlines from various North American and European cities, showing true variability effects [Pencept86e] Pencept "Software Toolkit for the Penpad 310 and Penpad 320", Pencept Division of Numonics, 39 Green Street, Waltham, Massachusetts 02154, 1986 Handwriting recognition tablet. Covers features and architecture of keyboard/tablet/mouse driver for PC- DOS/MS-DOC BIOS, control commands, form definitions, command templates, RPC calls (via serialized commands), stylus tilt angle, side-button, recognition macros (gesture commands), virtual tablet areas, PENPUT and PENGET for alternate input buffers (keyboard/recognition versus mouse/tablet/pointing). [Pencept86f] Pencept "Pencept PenPad with PenCAD", Pencept Division of Numonics, 39 Green Street, Waltham, Massachusetts 02154, 1986 Pencept PenPad interface for CAD systems: described as unique form of digitizer tablet with handwriting/printing character recognition. [Pencept86g] Pencept "Penpad Penpad 300 (TM) Digitizing Tablet Product Information", Pencept Division of Numonics, 39 Green Street, Waltham, Massachusetts 02154, 1986 Electromagnetic grid tablet, no handwriting recognition. [Pencept86h] Pencept "Introducing PenCad -- A Software Interface that puts CAD on a Pad", Pencept Division of Numonics, 39 Green Street, Waltham, Massachusetts 02154, 1986 PENCAD -- CAD templates and gestures ("recognition macros") for tablet with AutoCAD, other CAD systems. [Pencept86i] Pencept "PenPad Digitizer and Intelligent Tablets -- Using Artificial Intelligence to Make Your Computer Faster and Easier to Use", Pencept Inc., 39 Green Street, Waltham, Massachusetts 02154, 1986 Pencept PenPad 300, 310, 320 electromagnetic tablets, handwriting recognition, command macros (gestures) [Pencept86j] Pencept "Penpad 310 User's Guide", Pencept Division of Numonics, 39 Green Street, Waltham, Massachusetts 02154, 1986 (hardcopy manual) Handwriting recognition tablet, compatible with CalComp, Summagraphics, etc. On-tablet templates. [PennywittKE86a] Pennywitt, Kirk E. "Robotic Tactile Sensing", BYTE magazine, 1986, p. 177-ff Overview of tactile/touch sensors robotics: pressure sensing, slip (shear) sensing, temperature sensing. In particular PVDF polyvinyldene flouride piezoelectric polymer (elastomeric) over array of circular electrodes: compare with Boie 1974 multi-touch touchpad sensor. Also 16x16 array of anisotropic elastomeric conductor line arranged in grid, each of 256 contact point could be measured separately for pressure (more pressure == greater area of contact). Multiple grid-sensing architectures, some with high resolution. [Perex86a] Perex Ltd "Multipad from Perex", Perex Ltd., Reading, UK, 1986 Product literature on Perex Multipad Model 7500 portable (and desktop) touchpad digitizer data entry station: pre-printed forms with data entry icons (or virtual keyboard) recognized by bar code, use inputs data by taping/touching on form. Paper forms then usable as back-up physical record of data. One-line 40/80 character display. Compare with DataPad, which also had handwriting recognition. [PetersonJL86a] Peterson, James L. "A note on undetected typing errors", CACM, Vol 29, 1986, pp spelling correction using word lists compiled from published dictionaries, how to detect miss-typed/spelled word is another known word. Gives statistics for undetected errors, but a bit vague on how correctness was specified. [Pickering86] Pickering, J.A. "Touch-sensitive screens: the technologies and their applications", International Journal of Man-Machine Studies, Vol 25, 1986, pp Tutorial/review of tablet digitizer/touchscreen technologies that work with a finger touch, integrated with a display. Parallax error in touch-screen model. Capacitive, SAW acoustic surface wave, conductive/resistive film, infra-red optical (using shadow interpolation), surface reflection, strain gauge mounted overlay, velocity-based piezoelectric touch sensing. Mentions Knowlton visual keyboard as touch-input system. Eye dominance (parallax?), finger obscuring, use of stylus. Abstract device mapping (virtual tablets / virtual devices). Describes limitation of not detecting simultaneous multiple touches (multi-touch). Minimum touch 8:26:26 PM]

135 ( ) History of Pen and Gesture Computing zone 22mm (ATC air traffic control). Audio feedback ("beep") to confirm touch. Mack89 on tablet technology: [PlamondonR86a] Plamondon, R. and Baron, R. "On-line Recognition of Handprint Schematic Pseudocode for Automatic Fortran Code Generation", Proceedings of the 8th International Conference on Pattern Recognition, October 27-31, 1986, Paris, pp Graphical program editing user-interface using DCR [PlamondonR86b] Plamondon, R. and Baron, R. "A dedicated microcomputer for handwritten interaction with a software tool: system prototyping", Journal of Microcomputer Applications, Vol 9, 1986, pp On-line character recognition for graphical software tool User-interface goal: must interface to application without changing application Too much feedback distracts the user from the writing task [Postl86] Postl, W. "Detection of Linear Oblique Structures and Skew Scan in Digitizer Documents", Proceedings of 8th International Conference on Pattern Recognition, Paris, France October 1986, pp Cited in Biard86a [PoujoisR86a] Poujois, Robert "Capacitive Keyboard Structure", United States Patent 4,614,937, 1986 touch-sensitive (proximity) capacitive keyboard, wiring on opposite sides of substrate, describes matrix wiring of keys as to reduce connections: multi-touch [Quest86a] Quest Automation "Datapad puts your pen on-line to a computer", Quest Automation 1986 Handprinting recognition data entry terminal: uses two layers of conductive/resistive film, stylus presses together, floating handrest to keep hand from pressing on touch tablet. 40-character LED display. Application show user identifying document by writing in ID number, along with data. [RadioElectronics86] Radio Electronics "Equipment Report - Pencept Penpad 320", Radio Electronics, January 1987, pp 22 ff What a nice thing Pencept recognition macros (gestures) are for a user-interface [ReiffelL86a] Reiffel, Leonard: Jung, Wayne D. "System for Sensing Spatial Coordinates", United States Patent 4,603,231, July 29, 1986 capacitive/resistive sheet tablet, stylus is receiver. Generate AC voltage across conductive (resistive) sheet in X and in Y: measure strength of signal with gain correction for height gain correction used to determine Z proximity as well. Can run X and Y signals simultaneously if different frequencies, sheet may have nonuniform resistance with correction map. Can use more than one sheet, can be any number of surfaces. Gain for height is determined sum of signals measured with voltage from left-right and right-left. Compare with Turner paper on electric field sensing from Compare with ScripTel at Wang. [Rhyne86] Rhyne, J.R. and Wolf, C.G. "Gestural Interfaces for Information Processing Applications", IBM Research Report RC (Log #54544) 9/2/86, 1986, IBM T.J. Watson Research Center, PO Box 218, Yorktown Heights, NY Description of IBM's early informal study on what editing marks subjects use on spreadsheets Report on IBM's prototype gesture/handwriting Lotus interface Early description of IBM's electronic ink hardware tablet and display Refers to "gesture language" for the qualitatively similar set of symbol marks and operations subjects use to edit spreadsheets Refers to gesture language, combining gesture (pointing) and menu selection, gesture (command) and handwriting (data) Accuracy of recognition has great effect on user interface (need to confirm, not just correct) Recognition errors/accuracy: user assumes his or her writing was correct, so errors and means to correct must be very obvious For a small gesture symbol set, intra-user variability was found to be small For a small gesture symbol set, inter-user variability was usually three or so "qualitatively similar" (ambiguous/confuseable) symbols IBM had no answer on reconciling small display character size with larger handwritten character size for electronic ink User interface for electronic ink: enlarge area of interest, let user write handwritten characters normal size on enlarged image Optical/visual parallax, also ranging parallax (pen tilt) Pen tilt error of 0.06" tablet digitizer accuracy led to pointing errors and slow user interaction rate effects on user interface Handwriting substantially worse on slick glass surface than on paper for electronic ink, much lower recognition accuracy User interface problems for handwriting are big: no obvious ideas on how to solve [RichmondD86a] Richmond, Dana "Computer Terminal", United States Design Patent 283,026, March 18, 1986 Design of PenCept PenPad 200 handwriting terminal, showing digitizing tablet in front of a VT-100 like display with no keyboard / keyboardless [RothfjellRE86a] Rothfjell, Rolf E. "Method and Device for Signature Verification", United States Patent 4,581,482, April 8, 1986, assigned to Esselte Security Systems AB, Stockholm, Sweden. Signature verification using a constrained writing form to get alignment points for segmentation (!) [SabourinR86a] Sabourin, R. and Plamondon, R. "Preprocessing of Handwritten Signatures from Image Gradient Analysis", Proceedings of the 8th International Conference on Pattern Recognition, October 27-31, 1986, Paris, pp Static (OCR) signature verification system: test protocol included no forgery attempts [ScheiflerRW86a] Scheifler, Robert W. and Gettys, Jim "The X Window System", ACM Trans. on Graphics, Vol 5 8:26:26 PM]

136 ( ) History of Pen and Gesture Computing No 2, April 1986, pp Also as Internal Report, Laboratory for Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, October 1986 Description of the X-Window graphical user interface system. Pointing input devices include touchscreens, tablets, lightpens. See Gettys later work on RandR rotating windows extension in X windows. Transparent windows (drawn in parent window, so a sub-window?), but only as fully transparent and thus a clipping region for drawing: says other mechanisms usually better, no details. Brief mention of mouse/tablet/joystick/etc. input. Device independence without in GKS-style query for capabilities of devices. [SchellRR86a] Schell, Roger R. and Denning, Dorthy E. "Integrity in trusted database systems", Proc. 9th DoD/NBS Computer Security Conference, Gaithersburg, MD, September 1986 Discussion of DoD Trusted Computer System Evaluation Criteria (DOD 85). Trusted computer system, not only about secrecy (most security literature) but also integrity (control of unauthorized update/modification). Integrity: accuracy/correctness/validity. [SchomakerL86a] Schomaker, L.R.B. and Thomassen, A.J.W.M. "On the Use and Limitation of Averaging Handwriting Signals", in in "Graphonomics: Contemporary Research in Handwriting", Elsevier Science Publishers B.V. (North-Holland), 1986, pp Low-pass average filter on digitizer data so smoothed, but with anchored endpoints where velocity is zero Describes anchoring inflection (zero velocity) points on handwritten characters as similar to dynamic time warping for speech recognition Time-normalization of cursive handwriting; Force/pressure sensitive pen used in handwriting research; Capacitive coupling, resistive-sheet digitizer tablet [SchultzJ86a] Schultz, Jan "A history of the Promis technology: an effective human interface", Proc. ACM Conf. on The history of personal workstations, 1986, pp Promis: Early Digiscribe SAW touch-screen user interface system for medical records, 1967, user interface mostly point-and-click [ScriptWriter86a] Beasely, Richard "ScriptWriter personal correspondence, Tom Beasely of Data Entry Systems regarding PHIGS standards", Personal correspondence, 1986 Discussion of joint presentation to PHIGS graphical standards body concerning multi-modal input from handwriting recognition tablets from PenCept, CIC: gesture input with associated gesture/text character and positional information, compare wit multi-touch. Peter Bono presentation to PHIGS body. [Scriptel86] Scriptel "SPC-Series Technical Report", Scriptel Corporation, 4145 Arlingate Plaza, Columbus, OH 43228, May, 1986 Capacitive/electrostatic sheet digitizer tablet [SearbyAD86a] Searby, A.D. and Bowman, D.W. "Stylus Devices Responsive to Pressure Changes for Use in Videographic and Like Apparatus", United States Patent 4,580,007, April 1, 1986, assigned to Quantel, Limited, Surrey, England. Force/pressure-sensitive digitizer stylus with improved life and performance [SearbyAD86b] Searby, Anthony D. "Video Processing System for Picture Rotation", United States Patent 4,611,232, September 9, 1986 Rotation of digitized video image (in frame buffer) by bit-shift operations in real time. [Sejnowski86] Sejnowski, Terrence, J. and Rosenberg, Charles R. "NETtalk: A Parallel Network that Learns to Read Aloud", Technical Report JHU/EECS-86/01, John Hopkins University, Baltimore, MD, 1986 Hinton diagrams; training input consisted of phonetic transcriptions of English text, not actual spoken recording. [Shneiderman86a] Shneiderman, Ben "Human-Computer Interaction Research at the University of Maryland", SIGCHI Bulletin, January 1986, Vol 17 no 3, pp Published version of 1985 manuscript. User-interface studies on menu structure: globally visible menu helps User-interface: arrow keys for pointing vs mouse User-interface: manuals are better than on-line help (why?) User-interface: user preference for touch-screen pointing tablet/display over keyboard [Shridhar86] Shridhar, Am. and Badreldin, A. "Recognition of Isolated and Simply Connected Handwritten Numerals", Pattern Recognition, Vol 19 No 1, pp 1-12, 1986 Refers to stroke connection within and between characters [SibertJL86a] Sibert, John L.; Hurley, William D.; and Bleser, Teresa W. "An object-oriented user interface management system", Proc. SIGGRAPH '86, pp GWUIMS George Washington University User InterMace Management System: early object-based UIMS, message passing as buffered messages (?) rather than as method calls: virtual buttons, direct manipulation. Touchscreens tablets and mouse, character recognition (gesture recognition: compare early Alan Kay video). Franz LISP on Sun Microsystems workstation. [Signify86] Signify Inc. "Sign-On product information", 9005 Red Branch Road, Columbia Maryland 21045, 1986 (see also ATI). Signature verification product: now called ASI/Autosig Systems Inc.. 8:26:26 PM]

137 ( ) History of Pen and Gesture Computing [Srinivasan86] Srinivasan, S., Palaniswamy, K., and Natarajan, A.E. "Machine Recognition of the Indian Language Characters Using a Tree-structure Based On Primitives", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp [StefikM86a] Stefik, M; Bobrow, D.G.; Lanning, S; Tatar, D; and Foster, G. "WYSIWIS Revised: Early Experience with Multi-User Interfaces", Proc. CSCW '86, December 3-5, 1986, Austin Texas, pp Principles of whiteboard/chalkboard/blackboard systems: What You See Is What I See. Boardnoter freestyle sketching, erase gesture (not a shape) selects whole object not just pixels; with multiple users, visual clutter of windows a problem in the user interface, rooms and doors instead. Multi-touch? [Suen86] Suen, C.Y. "Human Recognition of Handprinted Characters and Distance Measurements", in "Graphonomics: Contemporary Research in Handwriting", Elsevier Science Publishers B.V. (North-Holland), 1986, pp Needed 29x39 resolution to get decent handwriting recognition Simple (not serious) recognizer algorithm using pixel template matching, then entropy, information content, and nearest neighbor distances: did not correlate to human recognition Human use multiple distance measurements (multiple features) to recognize characters Says entropy, hamming distance, linear and cross correlation, centre of gravity, Mahalonobis distance of template matching were all poor distance functions [Summagraphics86a] Summagraphics "Bit Pad Two Technical Reference", Summagraphics Corporation, August 1986 User's manual for Summagraphics Bit Pad Two (magnetostrictive?) tablet digitizer. Described proximity /out-of-prox position sensing, absolute coordinate mode and relative (mouse) mode. Both stylus and puck, stylus is an inking pen stylus. [Summagraphics86b] Summagraphics "SummaSketch: The world's best selling graphics Tablet", Summagraphics Corporation, 1986 SummaSketch: 6x9, 12x12, 12x18 line of electromagnetic tablets. Puck ("cursor") or stylus with side button, target is drawing market. [Suydam86] Suydam, W.E. "Approaches to Software Testing Embroiled in Debate", Computer Design, Vol 25 No 21, November 15, 1986, pp Long, informal review article on software testing and reliability David Parnas: Only actual use will bring out the mistakes (not testing) Parnas: need for regression testing Parnas: problem of classical testing is unweighted likelihood of failure Parnas: testing must consider seriousness of different types of failure Parnas: testing must differentiate between important errors and insignificant errors Parnas: (Titanic effect) software design implements a mental model of real world: must test the model AI testing: people think it's different from software testing: it is not AI testing: when mechanical engineering was a new as AI, had not invented the right angle yet Dijkstra: testing shows presence of bugs, not absence Testing: must distinguish between active failure (real bug) vs passive (design omission) Testing: only field testing finds "real" errors Testing: unanticipated data can seriously screw a program [SwinehartDC86a] Swinehard, Daniel C.; Zellweger, Richard J. Beach; and Hagmann, Robert D. "A Structural View of the Cedar Programming Environment", ACM Trans. on Prog. Languages and Systems, Vol 8 No 4, October 1986, pp Cedar OS at Xerox PARC. Cedar language is prototype-based (?) object-oriented language, garbage collection / automatic storage management, reference types instead of pointers, tiled window management. Graphics tablet and mouse both supported: single serial buffer of time-stamped input events. Mentions windows/objects with transparent colors (alpha blending), complains about the lack of transparent color support on system. [TagushiY86a] Tagushi, Y., and Yamanami, T. "Position Detecting Apparatus", United States Patent 4,617,515, October 14, 1986, assigned to Wacom Company, Limited, Japan. Cordless stylus digitizer tablet with magnets in stylus over a grid of wire loops [TanakaT86a] Tanaka, Toshinori, and Kobayashi, Shunsuke "Entry of Data and Command for an LCD by Direct Touch: An Integrated LCD Panel", SID 86 Digest of Technical Papers, Society for Information Display, 1986, pp Touch-entry-device integrated tablet/display using deformation from touch force/pressure on LCD pixel cells: pixels degrade, but refreshing fixes them. Capacitance between two grid layers of ITO on top and bottom of LCD pixels. [TanieK86a] Tanie, K.; Komoriya, K.; Kaneko, M.; Tachi, S.; and Fujikawa, A. "A High Resolution Tactile Sensor", In "Robot Sensors", Vol 2. Tactile and non-vision sensors pp , Springer Verlag, 1986 Touch sensor (touchpad) using array of phototransistors detecting light changes from total internal reflection: compare with FTIR by Jeff Han. Detects pressure distribution / 3D profile: multi-touch. Cites to Boie [TannerPP86a] Tanner, Peter P.; MacKay, Stephen A.; Stewart, Darline A.; and Wein, Marceli "A Multitasking Switchboard Approach to User Interface management", Proc. SIGGRAPH '86, Vol 20, No 4; pp :26:26 PM]

138 ( ) History of Pen and Gesture Computing User interface management system: refers to multiple virtual devices implemented on a tablet, applications need not know that the devices are virtual. [TanzawaS86] Tanzawa, Setsu "Recording pen", Japanese Patent Application Number S Light-sensing optical stylus digitizer for handwriting input, using X and Y lines in different colors. Compare with Anoto. [TeulingsH86a] Teulings, H., Thomassen, A.J.W.M., van Galen, G.P. "Invariants in Handwriting: The Information Contained in a Motor Program", in "Graphonomics: Contemporary Research in Handwriting", Elsevier Science Publishers B.V. (North-Holland), 1986, pp Spatial characteristics are more invariant in handwriting than time or force characteristics (? for signature verification?) But because handwriting is "efficient", time characteristics in handwriting are invariant on trained sequences, like signature verification [TeulingsH86b] Teulings, H., Mullins, P.A., and Stelmach, G.E. "The Elementary Units of Programming Handwriting", in "Graphonomics: Contemporary Research in Handwriting", Elsevier Science Publishers B.V. (North- Holland), 1986, pp Handwriting data low-pass filtered from 16 Hz to 48 Hz frequency There is no one, single grapheme unit of handwriting: units depend on what the subject is writing [TitusJ86a] Titus, Jon "Technology Update: Digitizing tablets offer choices of formats, operating modes, and pointers", EDN Electronics Design News, April 17, 1986, pp Report on digitizing tablets: mentions dead zone / minimum drag distance (press and hold?), mechanical joint digitizers, sonic 2D and 3D tablets/digitizers, formats. Stylus/puck referred to as "pointer". Vendors include Hewlett-Packard HP, SAC Science Accessories, Scriptel, Summagraphics, Tectronix, Numonics, Hitachi, Aristo Graphics, Altek, Calcomp, GTCO. Stream, point, poll, switched-stream modes. [TognazziniB86] Tognazzini, Bruce "You Had To Be There", report on User Interface Reference Model workshop at SIGCHI '86 conference, SIGCHI Bulletin, Vol 18 No 2, October 1986, pp Uses example of "graphical ambiguity/context" in recognition when discussing user-interface issues [Treisman86] Treisman, A. "Feature and Objects in Visual Processing", Scientific American, November 1986, pp 114B-125 Visual recognition consists of low-level features (automatic), then identification Low-level perception is lines, spots, edges, locations, orientation Properties/features conjoined only at higher level of human recognition (context): "distractors" delay human recognition [VanGalen86] Van Galen, G.P., Meulenbroek, R.G.J., Hylkema, H. "On the Simultaneous Processing of Words, Letters and Strokes in Handwriting: Evidence for a Mixed Linear and Parallel Model", in "Graphonomics: Contemporary Research in Handwriting", Elsevier Science Publishers B.V. (North-Holland), 1986, pp 5-20 Parsed characters in script handwriting into character segments at velocity zeros (sharp corners) Gives maximum velocity of handwriting in cursive script as 11 cm/sec Alternated turning direction vs continuous loops has big impact on handwriting velocity (of course, silly!) Beginnings of words written at slower speed [Wakahara86] Wakahara, Toru "personal correspondence", July 28, 1986 Personal correspondence, ambiguous pairs of characters in Japanese Katakana, Kanji. Transparent Electromagnetic digitizer in front (?) of flat display. [WardJR86a] Ward, J. "Method and Apparatus for Removing Noise at the Ends of a Stroke Caused by Retracing", United States Patent 4,608,658, August 26, 1986, assigned to Pencept Re-trace elimination to pre-process handwritten character shapes from tablet before recognition [Watanabe86] Watanabe, Y. "letter to J.Ward", Author's address: Terebijon Gakkaishi, Hachinohe Institute of Technology, Department of Engineering, Hachinohe, Japan, 25 November, 1986 Mentions co-work between Watanabe and Jiro Gyoba Interchange of looping and cusping is seen frequently in writing Japanese Kanji characters written to OCR standard are LESS human-recognizable than "sloppy" writing At msec/stroke, writing time for Kanji Japanese slower than Roman Constraints: Japanese have characters they can read, but do not know how to write Japanese writers do not conform to formal writing constraints in practice [Websters86] Merriam-Webster Inc "Moon Type", Webster's Third New International Dictionary of the English Language Unabridged, Copyright 1986, ISBN Show Moon Type, a simplified single-stroke / Unistroke alphabet used as embossed printing for the blind, popular before Braille became dominant [Wikipedia86a] Wikiepedia "Macintosh Programmer's Workshop", Wikipedia.org, fetched April 2010 Macintosh Programmer's Workshop MPW SDK/IDE, , supplanted by Project Builder IDE which became Xcode. SDK, not sure what it has for building applications: describes a special version of MAKE. [Wolf86a] Wolf, C.G. "Can People Use Gesture Commands?", SIGCHI Bulletin, Vol 18 No 2, pp Gesture input - user interface. IBM simple usability study (18 subjects) for "move" gesture consisting of an arrow to indicate what gets moved to where, with different gestures (lasso, bracket, underline) to indicate selection. Describes editing gestures, such as cross out gesture to delete a word, for stylus-based devices. 8:26:26 PM]

139 ( ) History of Pen and Gesture Computing [Wolf86b] Wolf, C.G. "Can People Use Gesture Commands?", IBM Research Report FC (#53105), 4/17/86 Gesture input - user interface. IBM simple usability study (18 subjects) for "move" gesture consisting of an arrow to indicate what gets moved to where, with different gestures (lasso, bracket, underline) to indicate selection. Describes editing gestures, such as cross out gesture to delete a word, for stylus-based devices. [WuL86] Wu, L. and Weng, F. "Chain Code for a Line Segment and Formal Language", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Some properties of chain codes (straight line theorem) very hard to test [YamamotoK86a] Yamamoto, K., Yamada, H., Saito, T. and Sakaga, I. "Recognition of Handprinted Characters in the First Level of JIS Chinese Characters", Proceedings of the 8th International Conference on Pattern Recognition, Paris, October 1986, pp Relaxation method with 98.6% success on Kanji/Chinese characters Testing with ETL-8 and ETL-9 "standard" Chinese writing samples in Japan ETL-9 standard Kanji Chinese samples is samples of 3036 characters [YamamotoK86b] Yamamoto, K. and Saito, T. "Pattern Reading System", United States Patent 4,566,124, January 21, 1986, assigned to Agency of Industrial Science and Technology, Ministry of International Trade and Industry, Tokyo, Japan. Handwriting character recognition by tracing contours, local extrema/outermost points, chain codes Template matching on direction chain codes, polygonal approximation to pattern for OCR handwriting recognition [Yau86] Yau, S. S. and Tsai, J.J.-P. "A Survey of Software Design Techniques", IEEE Trans. Software Engineering, Vol SE-12 No 6, June 1986, pp General survey article on software engineering biblio85 (Prev page) Main Page (Next page) biblio88 In many situations in life and work, standing your ground is generally a bad strategy: it just makes it easier for people to maneuver around you. "Nach seiner Weltanschauung, lebt er in einer bunten Welt wo alles schwarz oder weiss ist." 8:26:26 PM]

140 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1987 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:33 EDT Mirrors of this page can be found at: (Generally most up-to-date) biblio86 (Prev page) Main Page (Next page) biblio90 [AEG87] AEG Aktiengesellschaft "Polyform: The Future Data Entry Workstation", product literature, AEG Information Systems, Postfach 2154, D-7750 Konstanz, West Germany, 1987 AEG's OCR product for typed and handwritten character recognition [AbramsMD87a] Abrams, Marshall D. and Jeng, Albert B. "Network Security Protocol Reference Model and the Trusted Computer System Evaluation Criteria", IEEE Network Magazine, Vol 1 No 2, April 1987 pp Review of applying "trusted system" standards by DoD (Red Book, 1985) to network security. Cites to other on (mathematically-defined) formal security models. Authentication/identification of all network entities, Mandatory access controls based on sensitivity of information, discretionary access control like ACLs, Communications integrity, Service availability, accountability (logging). [AdlerR87a] Adler, Robert "Touch control system for use with or having a three-dimensionally curved surface", United States Patent 4,642,423, February 10, 1987 Sonic/acoustic wave digitizer/touchscreen for curved monitors: takes into account great-circle distance across curved plate, sonic transducers only at corners. Several references to Raleigh wave, elastic waves, infrared light-beam touch screens. Uses regular faceplate of monitor. [Adobe87a] Adobe "Adobe Illustrator User's Manual Ver. 1.0", Adobe, Inc., Jan 1987 Includes GUI with graphical elements. Gestures/Pop-up menu or keyboard: Cited in "Issues for locationindependent interfaces" [AhmedP87a] Ahmed, pervez and Suen, C.Y. "Computer Recognition of Totally Unconstrained Handwritten ZIP Codes", Int'l. Jnl of Pattern Recognition and Artificial Intelligence, Vol 1 No 1, 1987 Static/OCR recognition of handwritten zip codes. Segmentation as separate step before and independent of feature extraction. Primary features are end point direction, holes and enclosed curves. Tates 84% to 98% correction rate, but does not state whether per-character or per-zipcode, or false positive vs. rejects. [Anatex87] Anatex "Personal Writer Product Description", available from Anatex S.A., Xavier Maury, President, 18 Rue Troyen, F Paris, France, 1987 (see also "Personal Writer"). Adaptive handwriting recognition product from Anatex, France. In training procedure, if certain characters too similar (e, l, m n), tells user to change writing style. See also PersonalWriter89: company changed 8:26:29 PM]

141 ( ) History of Pen and Gesture Computing name. [AndersonPR87a] Anderson, Patrick R. and Winfree, L. Thomas jr., Eds. "Expert Witnesses: Criminologists in the Courtroom", SUNY Press, 1987 (hardcopy book) Collection of sociological papers on the experiences and roles of expert witnesses in criminal cases (not directly on the legal aspects of expert witnessing) [ArnautLY88] Arnaut, Lynn Y and Greenstein, Joel S. "Human factors considerations in the design and selection for computer input devices", Computer graphics: technology and applications, Academic Press, Boston, 1988 ISBN touchscreens: capacitive, resistive, crossed-wires, acoustic (using same elements for sound generation and detection): force (pressure, torque, and X/Y values from strain gauges at sides of touchscreen. Shows virtual slide and on/off switches circa 1985 on touchscreen. Touch tablets: elographics with stylus. Cites to Buxton for virtual input devices on a tablet. [AsboEL87a] Asbo, E.L., and Tichenor, H. "Method and Apparatus for Dynamic Signature Verification", United States Patent 4,646,351, February 24, 1987, assigned to Visa International Service Association, San Mateo, California. Signature verification using tip force (a.k.a. pressure), and length and angle of segments [AuerCM88a] Auer, Carol M.; Castagno, Daniel L.; Haley, Allen W. Jur.; Moore, Harry H. IV; O'Leary, Sean E.; Paley, Steven J; and Rutt, Thomas E. "Computer Interface Device", United States Patent 4,725,694, February 16, 1988 Slate-format/tablet-format touchscreen/pen computer with simulated keyboard on display. Keyboard/keypad layout, and labeling/function of keys can be changed dynamically. Return key removes virtual keyboard from display. Compare with Linus WriteTop? [BTG87] British Technology Group "Handwriting encoder and software for automatic signature verification", "Handwriting encoder may provide the access key for future systems users", Report CR , 101 Newington Causeway, London SE1 6BU, 1987 Combined pen and smart-card in 1987: invented by Colin Hilton, inventor: British Technology Group advisor: Dr. John Parks. Handwriting input/recognition system available for license from England [Baird87] Baird, H.S. "Feature Identification for Hybrid Structural/Statistical Pattern Classification", Laboratory report, 24-Feb Author's address: AT&T Bell Laboratories, 2C-557, Murray Hill, NJ Reports on OCR system (with Pavlidis) Training speed "very slow" on a statistical OCR recognition system Adaptive recognition: including "unrecognizable" characters (too small) aided training on normal size OCR Adaptive recognition: very large-scale trials necessary to determine actual performance Statistical recognition: in practice, "hand-crafted" (cognitive/functional) rules will always be needed as well Good 99%+ OCR results, but only for "excellent quality" input [Baptista88] Baptista, G. and Kulkarni, K.M. "A High Accuracy Algorithm for Recognition of Handwritten Numerals", Pattern Recognition, Vol 21 No 4, 1988, pp Multi-layered system syntactic and deterministic (feature table driven) approach to handwritten O.C.R. numeral recognition based on human visual cortex Criticizes syntactic character recognition: says it is unwieldy, and cannot be used in a trainable system Diagram with hierarchy of human visual processing, from cells to brain stem to relations O.C.R. using noise removal, thinning, internal segmentation (like chaincodes) [BarkerBA88] Barker, B.A., Hernandez, I.H., and Machart, B.H. "Integrated multiple data editor", United States Patent 4,723,209, February 2, 1988, assigned to IBM Corporation. Integrated multiple data editor uses super-block structure of objects sets and flow attributes for text object sets [BenaryJ88a] Benary, J. "Attributed Grammars and Two-Dimensional Formal Languages", Technische Universit&aumlt Dresden, 1988 Cited in Marzinkewitsch [BierEA88a] Bier, Eric A. "Snap-Dragging: Interactive Geometric Design in Two and Three Dimensions", UC Berkeley Report UCB/CSD , May 1988 Snap to grid vs. heuristic constraints in three dimensions 3-D similar to compass and ruler for pointing and positioning, in technical illustration system. Slope, angle, distance constraints for lines, circles, planes and spheres (alignment object) specified by user. [Blatt88] Blatt, Louis "Character Recognition and Added Value", internal memorandum, Wang Laboratories, September 27, 1988 Review of user-interface human-factors ergonomics advantages/disadvantages in a portable handwriting recognition product: just a rehash of Wolf87 on gestures. User interface: ease of learning may be more important than ease of use. [Blesser87] Blesser, B. "Position Indicating Apparatus for Use in a Digitizing Tablet System", United States Patent 4,638,119, January 20, 1987, assigned to Pencept, Incorporated, Waltham, Massachusetts. 8:26:29 PM]

142 ( ) History of Pen and Gesture Computing Pencept digitizer/tablet patent: two-coil pen stylus, side button, force (a.k.a. pressure) transducer [Blesser87a] Blesser, B., and Prentice, T. "Digitizing Tablet System", United States Patent 4,644,102, February 17, 1987, assigned to Pencept, Incorporated, Waltham, Massachusetts. Pencept digitizer patent [Bokser88a] Bokser, Mindy R. "Pattern Classification Means for Use in a Pattern Recognition System", United States Patent 4,773,099, September 20, , assigned to The Palantir Corporation, Santa Clara, California. Pattern recognition (for characters) using a set of "ringed zones" in feature space to define boundary space and confidence probability Contains interesting diagram for a discussion of one character overlapping/invading the feature space boundary of another. [Bokser88b] Bokser, Mindy R. "Means for Resolving Ambiguities in Text Based Upon Character Context", United States Patent 4,754,489, June 28, 1988, assigned to The Palantir Corporation, Santa Clara, California. Context in character recognition similar to di-grams, tri-grams, n-grams: aggregate probability score of all character matched against a dictionary of words [Booth87] Booth, D.S., Bryden, M.P., Cowan, W.B., Morgan, M.F., and Plante, B.L. "On the Parameters of Human Visual Performance: an Investigation of the Benefits of Antialiasing", unpublished manuscript, Authors' address: Computer Graphics Laboratory, University of Waterloo, Waterloo, Ontario Canada N2L 3G1 Quotes Allen Newell: engineering understanding in user-interfaces and human factors more important than psychological mechanisms [Borland88a] Borland International "SideKick Plus for SideKick Users: The Professional Desktop Manager", Borland International, 1988 Introduction to features of SideKick Plus over SideKick. Text editor, Time planner/calendar, telephone dialer for DOS desktop. Enhanced version of SideKick. Character-mode windows. Screen dialing: Can search display screen for telephone numbers. [Borland88b] Borland International "SideKick Plus: The Professional Desktop Manager -- Owner's Handbook", Borland International, 1988 User manual/guide for Sidekick Plus desktop PIM personal information manager for DOS. Phonebook database with automatic dialing. Quick-dial dials numbers at cursor position on screen. Phonebook automatically searches screen for phone number: regular Sidekick does this by default. (page 172) [BrassLB88] Brass, Robert L.; Glaberson, John; Mason, Richard W.; Santulli, Scott; Roth, G. Thomas; Feero, William M., and Blalaska, Richard K. Jr. "Method and Apparatus for Transforming Digitally Encoded Data into Printed Data Strips", United States Patent 4,728,783, March 1, 1988, assigned to Cauzin Systems, Incorporated Two-dimensional bar code printed with matrix printer, strip data can be arbitrarily long. Contrast to Sekendur patterns [Brault87] Brault, Jean-Jules and Plamondon, Rejean "Handwritten Curve Partitioning Based on Geometric and Sequential Information", Proceedings of the Third International Symposium on Handwriting and Computer Applications, Ecole Polytechnique de Montreal, Canada, 1987, pp Segmentation of handwriting using local extrema / points of high curvature [BrittsS87] Britts, Stefan "Dialog Management in Interactive Systems: A Comparative Survey", SIGCHI Bulletin, Vol 18 No 3, January 1987, pp Cites to 1986 Ward paper as example that UI / Dialog management systems for user interfaces do not accommodate new types of devices, such as handwriting recognition input. [Burgess87] Burgess, Ken L. "Digitizing Method and Apparatus for Compensating Cursor of Stylus Velocity Errors", United States Patent 4,686,331, August 11, 1987 Correction for bowing due to velocity on digitizing tablet using double alternating X and Y measurements [Buxton87a] Buxton, W. "Research Frontiers and Unsolved Problems", in Readings in Human-Computer Interaction: a Multidisciplinary Approach, Chapter 14, 1987 Gesture and character recognition: shorthand gesture user-interface for music. [Buxton87b] Buxton, William and Kurtenbach, Gordon "Editing by Contiguous Gesture: A Toy Test Bed", Computer Systems Research Institute, University of Toronto, Toronto, Ontario Canada M5S 1A4, 1987 Suggested symbols for gesture recognition operations: lasso, angles, directional strokes, single-line crossout. Combine lasso-drag-"c" gestures in one combined mark. Double lasso for selection with exceptions. [CCIT88a] CCITT International Telegraph and Telephone Consultative Committee "Draft Recommendation X.509: The Directory-Authentication Framework", CCITT International Telecommunication Union, November 1988 Draft standard for CAs certification authorities from ITU: includes introduction to RSA public key cryptosystems, definition of PKI certificates. [CIC88a] CIC "Handwriter (R) ProFicient (R) AutoCad Enhancer User's Guide / Proficient Series Software Drivers for AutoCad, Synthesis, NC Programmer, etc.", Communication Intelligence Corporation, 333 Ravenswood Avenue PN 257, Menlo Park, CA 94025, 1988 Gesture-like handwriting user interface for office automation and data entry from CIC. Compare with PenCAD offering by PenCept, Ledeen Recognizer. 8:26:29 PM]

143 ( ) History of Pen and Gesture Computing [CIC88b] CIC "Handwriter (R) WRITE-ON (R) Series Software Drivers for Microsoft Windows, WordPerfect, Pagemaker, etc.", Communication Intelligence Corporation, 333 Ravenswood Avenue PN 257, Menlo Park, CA 94025, 1988 Gesture-like handwriting forms data entry from CIC Tap different control areas to shift from upper to lower, etc. [CIC88c] CIC "Handwriter (R) Customization tools: FormManager (TM), FormMapper (TM), etc.", Communication Intelligence Corporation, 333 Ravenswood Avenue PN 257, Menlo Park, CA 94025, 1988 Signature verification software from CIC [CIC88d] CIC "Handwriter (R) On-line Dynamic Signature Verification (TM)", Communication Intelligence Corporation, 333 Ravenswood Avenue PN 257, Menlo Park, CA 94025, 1988 [CIC88e] CIC "Handwriter (R) Data Entry System, Layout Design System (TM) Manual Version 1.1, May 1988(TM)", Communication Intelligence Corporation, 333 Ravenswood Avenue PN 257, Menlo Park, CA 94025, 1988 Forms-design package for handwriting, using handwriting: separate writing areas for upper and lower case, numerics, Command templates (on-tablet menus), etc. File includes product description materials: Layout Design System for Customized Computer Application in Handwriter Series [CIC88f] CIC "Handwriter (R) Data Entry System User's Guide(TM)", Communication Intelligence Corporation, 333 Ravenswood Avenue PN 257, Menlo Park, CA 94025, 1988 Shows recognition macros / gestures, special shape for erase (inverted U): features copied from PenPad product [CIC88g] CIC "Handwriter (R) Write-On(TM) Form Manager User's Guide Version 1.0", Communication Intelligence Corporation, 333 Ravenswood Avenue PN 257, Menlo Park, CA 94025, 1988 User guide / installation guide for CIC HandWriter forms application software: shows command and digitizer input areas on tablet for forms-layout design program using HandWriter tablet with handwriting recognition. Field types/context in recognition: check box, date, fixed point, integer alphanumeric, etc. Command-line interface. [CIC88h] CIC "Proficient AutoCAD Enhancer", Communication Intelligence Corporation, 333 Ravenswood Avenue PN 257, Menlo Park, CA 94025, 1988 Template for AutoCAD on DOS/PC using handwriting recognition and pre-defined menus on the tablet, external to the application. Compare with Pencept CAD products. Electronics board noted as working with multiple third-party digitizer tablets. [CLCS88] Chinese Language Computer Society "Proceedings of 1988 International Conference on Computer Processing of Chinese and Oriental Languages", August 29 - September 1, 1988, Toronto, Canada. Handwritten and speech Chinese, Japanese/Kanji recognition [CallahanJ88a] Callahan, Jack; Hopkins, Don; Weiser, Mark; and Shneiderman, Ben "An Empirical comparison of Pie vs. Linear Menus", Proc. CHI '88, pp Pop-up pie menus: permits multiple motions, action is selected on mouse-up / end-of-stroke to confirm correct choice. Other references cite for T-Cube pie-menu unistroke handwriting (recognition) input from Apple. Press-and-hold (press-and-hold-still), pressing and holding with any of three mouse buttons to invoke pop-up menu. [CardelliL88a] Cardelli, Luca "Building User Interfaces by Direct Manipulation", Proc. UIST '88, pp Dialog editor for creating graphical UI user interfaces. Refers to buttons for rotations and reflections of bitmap image being constructed for an icon/cursor. [CarrollAB87a] Caroll, Arthur Bruce; Hough, Steward Edward; Hunter, Pual Richard; Carstedt, John Keith; Shaw, Sam Richmond, and Garrett, James Edward "Signal preconditioning for touch entry device", European Patent Application A1, October 7, 1987 Optical touchscreen, normalization of output signals (signal strength of optical emitter/detector pairs) [Casio87a] Casio "Casio PF-8000 Calculator (IF-8000)", Collection from Pocket Computing Museum, Calculator (this reference says 1987) with touchscreen input, and also zone-based handwriting recognition on a touch-film keyboard/keypad Same reference includes Kyocera Refalo (1991) Same reference includes Sony Magic LInk, General Magic OS Same reference includes EO Personal Communicator, 1994, PenPoint OS from GO Same reference includes Simon from IBM 1994: Phone built into a portable computer Same reference includes Nokia 900 communicator: fax/phone/browser Casio IF and PF-8000: same product? [Casio87b] Casio "Casio Handhelds Product Information (in Japanese)", and other sources, fetched 2002 Information on Nokia and other handhelds: Sharp PA-8500 (1988), Casio PF-8000 with handwriting recognition touchscreen/touchpanel (1984), Casio DK-5000 full keyboard (1990), HP 95LX full keyboard (1991), HP 100 LX, HP 200 LX, IBM Palm Top PC 110 (with touchpanel?) 1996, IBM ChipCard TC :26:29 PM]

144 ( ) History of Pen and Gesture Computing credit card format (1995), Sharp PA-M1 (1996), NTT DoCoMo Pocketboard (1998), NEC Mobile Gear MC-P1 (with Pen/GEOS?), Refalo, Casio PB-100, CVB-1000 touchscreen organizer watch, Casio IF-8000 touchscreen organizer with electronic ink on touchscreen, Casio NX-6000 / Planeo PDA, Sharp IQ-9200 with touchscreen PDA/organizer, Casio IT-2000, [CaswellNS88a] Caswell, N.S. "Introduction to input devices", Chapter in "Input Devices", Sol Sherr Ed. Academic Press, Boston, 1988 ISBN (electronic book) Cited in LCS/Telegraphics case vs. Schumer for input devices like virtual tablets Section "Digitizers and input tablets" by Thomas E. Davies, Gerard Mathews, and Paul D. Smith Coordinate transforms for calibration on digitizer tablet. Active surface tablets vs. Touch Screens. Refers to rudimentary touch screen composed of switches. Cites problems of acoustic, voltage gradient (resistive film), acoustic touch, multilayer matrix-encoded digitizer tablets. Touchpad integrated with keyboard. Comparison of mouse and tablet. see also Sherr88: check Barker Engineering Library. multi-touch capable? [CaswellNS88b] Caswell, N.S. "Introduction to input devices", Chapter in "Input Devices", Sol Sherr Ed. Academic Press, Boston, 1988 ISBN (electronic book) (See also SherrS88a) [ChiaoYueh88] Chiao-Yueh, Lin "Method and Means for Automatically Coding and Inputting Chinese Characters in Digital Computers", United States Patent 4,758,979, July 19, 1988 Kanji/Chinese character recognition using a special digitizer stylus Gesture-like handwriting user interface for AutoCad from CIC [ClarkWA87] Clark, Wesley A. "The LINC was early and small", History of Medical Informatics: Proc. of ACM conference on History of medical informatics, Bathesda Maryland, 1987, pp Also available at Description of LINC "personal" laboratory computer with DEC- Tape/LINC-tape. Graphical input for cursor crosshairs (without a mouse) using analog knobs on A/D converter module. Citations to TX-O and TX-2 computers (which ran Spacewar and Sketchpad) [ComerfordLD87a] Comerford, Liam "Proof of Concept Models: Seeing Eye Mouse", fetched 2016 Web posting by Liam Comerford Seeing Eye Mouse: mouse with single-cell braille display under fingertip for reading a (text) screen. Limitation was that Braille users must scan finger across cell to read. Refers to mouse gestures for swipe/scroll. [ComerfordLD88a] Comerford, Liam David and White, Steve Richard "Manipulating Rights-To-Execute in connection with a software copy protection mechanism", European Patent Application EP A3, May 25, 1988 Same as US Patent US DRM/licencing control for software: conditioned on number of executions, period of time, backup of software license encrypted. [ComerfordLD88b] Comerford, Liam "Proof of Concept Models: A Basic Yorktown Security System (ABYSS)", fetched 2016 Web posting by Liam Comerford on origin of (physically) secure systems for DRM digital rights management (epoxy encapsulation, etc.). Encapsulated/tamper-proof hardware single-use hardware token, key management. Project not supported by management. [Compucon87] Compucon Services Corporation "letter from Bruce D. Holenstein to Jean Renard Ward", 83 Main Street, Newton, New Jersey 07860, June 23, 1987 Letter from Compucon looking for somebody to buy their character recognition software: does OCR only: they don't admit it doesn't do handwriting [Cooper88] Cooper, Leon N., Elbaum, Charles, Reilly, Douglas L., and Scofield, Christopher L. "Parallel, Multi-Unit, Adaptive, Nonlinear Pattern Class Separator and Identifier", United States Patent 4,760,604, July 26, 1988, assigned to Nestor, Incorporated, Providence, Rhode Island. Sort of a neural-net recognizer for patterns not linearly recognizable [CraneHD88a] Crane, Hewitt D. and Ostrem, John S. "Process and Apparatus Involving Pattern Recognition", United States Patent 4,718,102, January 5, 1988, assigned to Communication Intelligence Corporation, Menlo Park, California Loosely-worded patent on a first pass recognition for handwriting (Kanji) that intentionally leaves in ambiguous categories (initial classification), then a second pass for disambiguation Described basic strokes, then a word/stroke dictionary for Kanji/Chinese recognition Contains review of commercial work in Kanji/Chinese recognition. Cited for Unistroke. [DataEntry88] Data Entry Systems "product information", 1988: see ScriptWriter. [DataSystems87a] DataSystems UK Ltd "The PAD", DataSystems UK Ltd, 35 Holly Avenue, Breaston Derbyshire DE7 3BG, United Kingdom AC plasma display integrated with high resolution transparent digitizer: digitizer type not stated. VME bus, tablet processor controller, 200 points/second, 0.001" nominal resolution. See also National Physical Laboratory. 8:26:29 PM]

145 ( ) History of Pen and Gesture Computing [DaviesTF88] Davies, Thomas F., Matthews, H. Gerard, and Smith, Paul D. "Digitizers and input tablets", "Input Devices", Sol Sherr Ed. Academic Press, Boston, 1988 ISBN , Chapter 4 (electronic book) (hardcopy book) (See also SherrS88a) Cited in LCS/Telegraphics case vs. Schumer for input devices like virtual tablets [DayBW88a] Day, Benjamin W. Jr, Gillon, Alexander C., and LeConte, Raoul A. "Touch Screen Form Entry System", United States Patent 4,763,356, August 9, 1988 User-Interface Form input with pen stylus, field highlighted as they are selected, also pops up appropriate GUI tool for that field: pop-up graphical/virtual keyboard, date pad, numeric pad, calculator, electronic ink drawing/writing field. No handwriting character recognition? [DhawanSK87a] Dhawan, Satish K. "Electrostatic Pattern-Coupled Digitizer", United States Patent 4,705,919, November 10, 1987 Charge-ratio electrostatic tablet digitizer [DiederichJ87a] Diederich, Jim and Milton, Jack "Experimental Prototyping in Smalltalk", IEEE Software, May 1987, pp Smalltalk IDE development environment: modeless development (rapid turnaround on code/execute, in this case via interpreter) speeds up development, object-based programming with strict encapsulation tends to make code messes be a high levels of abstraction, not low. Pluggable views, similar to MacApp IDE. [Digitalk88] Digitalk Inc. "Smalltalk/V 286 Tutorial and Programming Handbook", Digitalk Inc., Los Angeles California, 1988 Introduction to Smalltalk language and environment. (Electronic file). [Dilella87] Dilella, Antonio "Method and Apparatus for Isolating Image Data for Character Recognition", United States Patent 4,680,803, assigned to NCR Corporation, Dayton, Ohio, July 14, 1987 Separates hand-written characters on checks prior to recognition: segmentation by looking at joins, feedback from recognizer [Doran88] Doran, David "Trackballs and joysticks", Appears in Computer graphics: technology and applications, Academic Press, Boston, 1988 ISBN Cited in LCS/Telegraphics case vs. Schumer for input devices like virtual tablets [DruinA88a] Druin, Allison "NOOBIE: the animal design playstation", ACM SIGCHI Bulletin, Vol 20 No 1, July 1988, pp, Large stuffed animal designed to interact with children, respond to physical squeezes and touches with sounds. Mostly a study of making machines interactive: compare with later work on animated faces and teddy bears at M.I.T. [Dunkley88] Dunkley, Rowland A. and Pugsley, Peter C. "Sign Verification", United States Patent 4,752,965, June 21, 1988, assigned to The De La Rue Company, PLC, London, England. Portable signature verification system using a force/pressure-sensitive pad/tablet [El-Sheikh88] El-Sheikh, T.S. and Guindi, R.M. "Computer Recognition of Arabic Cursive Scripts", Pattern Recognition, Vol 21 No 4, 1988, pp O.C.R. system for Arabic script from typewritten (but cursive, like handwriting) characters Uses Fourier descriptors for characters, topological classifier for dots, stress marks [Ellozy88a] Ellozy, H.A. et al "The paper-like interface (video)", IBM internal video shown in office lobby circa 1989: web links in paper file Description of paper-like interface PLI projects at IBM: see refs. Video also shown at CHI89. [Etherington87] Etherington, H.J.C., Joslin, P.C., and Dunkley, R.A.G. "Sign Verification", United States Patent 4,680,801, July 14, 1987, assigned to The De La Rue Company PLC, England. Signature verification [EvansJW88a] Evans, John W. "Capacitance-variation-sensitive touch sensing array system", United States Patent 4,733,222, March 22, 1988 Capacitive grid touchscreen using diamond shapes on electrodes, separate determination at each node. Shows alternative spiral patterns instead of X/Y grid. Used for touch-sensitive keyboard, may also be used for touch sensing, or finger in "close proximity". Interpolation of multiple readings to get better position. Refers to transparent conductive material for grid so that touchscreen may overlay a CRT display. Conductor traces on opposite sides of board to avoid cross-over. multi-touch capable? [FeldmanJA88a] Feldman, Jerome A., Fanty, Mark A., Goddard, Nigel H., and Lynne, Kenton J. "Computing with Structured Connectionist Networks", CACM, Vol 31 No 2, February 1988, pp Overview article on neural networks: asserts merging connectionist/relaxation/adaptation AI/patternrecognition with algorithms+data structures/inference/representation would be good, but it's just speculation [Filliman87] Filliman, Mark D., and Granzow, Robert H. "Personal Identification Method and Apparatus", United States Patent 4,656,662, April 7, 1987, assigned to NCR Corporation, Dayton, Ohio. A patent on the idea of doing signature verification (of any means) using a lightpen and a display 8:26:29 PM]

146 ( ) History of Pen and Gesture Computing [FordDM87a] Ford, D.M.; Higgins, C.A.; Brocklehurst, E.R. "The Electronic Paper Project", Proc. 3 Int'l Symp. on Handwriting and Computer Applications, Montreal 1987, pp Report on paper-like interface / Electronic Paper project PAD. Free-hand editing using proofreader's marks gestures by BSI standard BS5261(1976) [Fox87] Fox, A.S., and Tappert, A.S. "On-line External Word Segmentation for Handwriting Recognition", unpublished manuscript, authors' address: Computer Sciences Department, IBM T.J.Watson Research Center, PO Box 704, Yorktown Heights, NY 10598, 1987 Asserts external segmentation (separate from recognition) is faster, more interactive Cites 0.5% segmentation error, but with six very careful writers Cites that sophisticated segmentation can be done using extra information from having boxes Asserts delayed strokes (t, i crossings and dots) always done at end of each word [Fox88a] Fox, A.S., Greanias, E.C., Kim, J., and Tappert, C.C. "System for Automatic Adjustment and Editing of Handwritten Text Image", United States Patent 4,727,588, February 23, 1988, assigned to International Business Machines Corporation, Armonk, New York. Scribble-editor: gesture recognition? user interface for text-editing/word-processing of handwritten annotation electronic ink Automatic editing and adjustment system for handwritten text images using processing algorithm to identify groups and to smooth random fluctuations in handwritten information [Fox88b] Fox, A.S., Kim, J., and Tappert, C.C. "Reco machine interfaces", IBM Research Report RC14053, September 1988 Tappert89, describing IBM's handwriting recognition [Fukunaga87] Fukunaga, Y., Kuzunuki, S., Shojima, H., Yokoyama, T., Koga, K., Hirasawa, K., and Kawada, S. "Apparatus for Recognizing and Displaying Handwritten Characters and Figures", United States Patent 4,641,354, assigned to Hitachi, Limited, Tokyo, Japan, February 3, 1987 User-interface: ink erasure for electronic ink and handwriting recognition Patent on electronic ink where handwritten strokes are erased automatically as the strokes for a character are recognized and the ASCII result is displayed instead [FulksRG87a] Fulks, Robert G. and Hager, Robert J. "Method and apparatus for sensing activity for a keyboard and the like", United States Patent 4,649,784, March 17, 1987 Force/pressure sensing for (musical/piano) keys of keyboard, using force/pressure sensitive ink layer to measure force. [FurukawaM88a] Furukawa, Mikio; Tahara, Kazatoki; and Kunishi, Yosuke "Touch-operated See-through Coordinate Input Unit", United States Patent 4,725,696, February 16, 1988 Resistive-film transparent digitizer using grid of conductors between spacer dots: multiple wires between dots for relable operation. Electroconductive ink or paint, or thin wires of metal. [GallantSI88a] Gallant, Stephen I. "Connectionist Expert Systems", CACM, Vol 31 No 2, February 1988, pp Connectionist / neural network overview Chinese input system using bar-coded "digitizer" and radical identification Chinese input with over 6000 characters (not Kanji/Japanese) [GerbEletronik87a] Gerb Elektronik "CHI-EASY Product Description", (in German and Chinese), Gerb Elektronik Berlin, Roedernallee , 1000 Berlin 51, West Germany, 1987 [GiuseD88a] Giuse, Dario "LISP as a Rapid Prototyping Environment: The Chinese Tutor", Lisp and Symbolic Computation, Vol 1, 1988, pp Lisp as rapid-prototyping language and UIMS for user interfaces [GoodmanD88a] Goodman, Danny "The Complete HyperCard Handbook, Second Edition", Bantam Books, 1988 HyperCard reference and tutorial: browser for forms and information connected by hyperlinks (within the set of hypercard stacks), scripts/calculations associated. [Goshtasby88] Goshtasby, A. and Ehrich, Roger W. "Contextual Word Recognition Using Probabilistic Relaxation Labeling", Pattern Recognition, Vol 21 No 5, 1988, pp Spelling context correction after recognition: does not use confidence figure from recognition about how good a character is, or what else came close Post-processing spelling context correction: examples show incorrect correction Spelling context correction using probabilities of letter pairs Describes relaxation process for spelling context correction by factoring in the merit figure for each character's % confidence level [GouldJD87a] Gould, J.D. and Salaun, J. "Behavioral Experiments on Handmarkings", Proceedings of CHI+GI Conference on Human Factors in Computing Systems and Graphics Interface, Toronto, April 5-9, 1987, pp IBM study of what gestures/symbols are most "natural" for user interfaces for editing drawing and text: data captured using Macintizer tablet, but no recognition of gestures by computer in system. Pigtail and "X" gestures for delete gestures, circle and arrow gestures for highlighting, arrow gesture for move, lasso gesture for select, etc. 8:26:29 PM]

147 ( ) History of Pen and Gesture Computing [GouldJD87b] Gould, J.D. and Salaun, J. "Behavioral Experiments on Handmarkings", A.C.M. Trans. Office Information Systems, Vol 5 No 4, October 1987, pp Sloppy IBM study of what gestures/symbols are most "natural" for user interfaces User interface / gesture recognition: "natural" symbols are circles for groupings, arrows for operators and target position User interface / gesture recognition: design of the actual user interface will actually determine whether gesture symbols are useful, not the use of gestures per se "natural" user-interface study of gesture symbols for text editing / word processing, picture editing [GouldJD87c] Gould, J.D., Alfaro, L., Finn, R., Haupt, B., Minuto, A., and Salaun, J. "Why reading was slower from CRT displays than from paper", Proceedings of the 1987 CHI+GI Conference: Human Factors in Computing Systems and Graphics, Toronto, Ontario, Canada, 1987 Graphic user-interface hardware: what makes a display easy to read [Goy88] Goy, Carl "Mice", Computer graphics: technology and applications, Academic Press, Boston, 1988 ISBN Cited in LCS/Telegraphics case vs. Schumer for input devices like virtual tablets [Graphonomics87] International Graphonomics Society "Membership directory", March 1987, c/o Department of Experimental Psychology, University of Nijmegen, PO Box 9104, 6500 HE Nijmegen, The Netherlands. Index of researchers in handwriting analysis, signature verification, and character recognition [Greanias87] Greanias, E.C., J.J., Verrier, G.F., and Donaldson, R.L. "Combined finger touch and stylus detection system", United States Patent 4,686,332, August 11, 1987, assigned to International Business Machines Corporation, Armonk, New York. Integrated transparent tablet/display for data processing. Flexible film to fit on curved CRT display. Stylus detection capacitive digitizer and finger touch touchpad in one device, integrated with a display. Capacitive(?) grid. Not clear that it can sense multiple simultaneous finger touch points (multi-touch), but can sense stylus and finger input simultaneously. Compare with Scriptel. Touch detected by strength of signal: 919 patent says this sometimes gave false touches due to variation in signal between grid lines. Bimanual/multi-touch. flexible ITO on flexible film, whole thing is transparent [GreaniasEC88a] Greanias, E.C., Schroeder, K.F., and Ruffino, L.V. "Minimum parallax stylus detection", United States Patent 4,764,885, August 16, 1988 Parallax error minimalization for display device / tablet digitizer - enables operator to focus attention on displayed location of cursor instead of stylus location. Emphasizes display of tracking cursor to remediate problems from parallax displacement of reported position from physical position of stylus. [GreensteinJS88] Greenstein, Joel S. and Muto, William H. "Keyboards", Computer graphics: technology and applications, Academic Press, Boston, 1988 ISBN Cited in LCS/Telegraphics case vs. Schumer for input devices like virtual tablets [Gundersen88a] Gundersen, Steven C. "Measure of Distinguishability for Signature Verification", United States Patent 4,736,445, April 5, 1988, assigned to International Business Machines Corporation, Armonk, New York. Signature verification where the most reliable criteria are applied first, then the less reliable criteria or similarity score. Uses X/Y acceleration and pressure/force. Several references to Hurst. [Gundersen88b] Gundersen, Steven C. and Worthington, Thomas K. "Signature Verification Algorithm", United States Patent 4,789,934, December 6, 1988, assigned to International Business Machines Corporation, Armonk, New York. Signature verification using spatial domain, then successively acceleration in frequency domain and force/pressure profile for a coherence score [HCI87] Advance Program "Second International Conference on Human-Computer Interaction", Honolulu, Hawaii, August, 1987 [HallAD88a] Hall, Anthony D., Cunningham, James B., Roache, Richard P., and Cox, Julie W. "Factors Affecting Performance Using Touch-entry Systems: Tactual Recognition Fields and System Accuracy", Journal of Applied Psychology, Vol 73 No 4, 1988, pp Study of minimal acceptable optical parallax, angle (standing/sitting, vertical/horizontal touchscreen), size of target on touchscreens/electronic-ink tablet: how much to offset the cursor affects usability. Bigger targets are easier to target. Cites to parallax problems on touchscreens. [HallAD88b] Hall, A.D., Cunningham, J.B., Roache, R.P. and Cox, J.W. "Factors affecting performance using touchentry systems: tactual recognition fields and resolution characteristics", Technical Report No TR , IBM Corporation, Research Triangle Park, North Carolina, 1988 Beringer89: integrated tablet/display hardware for user-interface? [Hamel88] Hamel, K. "NoteWriter computer software", Computer Software, University of British Columbia, Vancouver, Canada, Cited by Brown in 1998: unistroke music notation handwriting recognition system [HelanderM88a] Helander, Martin et al, Eds. "Handbook of Human Computer Interaction", Elsevier Science Publishers, :26:29 PM]

148 ( ) History of Pen and Gesture Computing Includes chapters Direct Manipulation, Graphics Challenges in Designing Object-oriented User Interfaces, Input Devices (Gesture-based input), Rapid Prototyping for User Interface Design (Wizard-of-Oz) [HellmanME87a] Hellman, Martin E. "Software Distribution System", United States Patent 4,658,093, April 14, 1987 DRM: One-time-use enabling codes for software. Can authorize number of times to use before needing new license authorization. Requires private key on each device for non-transferability. [Hernandez87a] Hernandez, I., Baeker, B.A., Himelstein, C.S. and Machart, B.H. "Business chart arm editing method", United States Patent 4,674,043, June 16, 1987, assigned to IBM Corporation, Armonk, New York. Business chart arm editing method updating predefined data to reflect result of editing action [Hernandez87b] Hernandez, Irene H., Himelstein, Carol S., and Wang, John S. "Method of Editing Graphic Objects in an Interactive Draw Graphic System Using Implicit Editing Actions", United States Patent 4,686,522, August 11, 1987, assigned to IBM Corporation, Armonk, New York. User-interface on graphics drawing/editing: one button to pull up a menu, a second button to select: mode of action persists until stopped User-interface: editing in handwriting/graphics is to modify (several steps in a row), not correct (one step) as in text editing [Herrndobler87] Herrndobler, I. and Machalitzky, C. "Schreiben mit Uli: Schreiblehrgang", Paul List Verlag, Munich, West Germany, 1987 German handwriting textbook to show national variation in handwriting styles (for script) [HigginsCA88a] Higgins, Colin Anthony; Leedham, Graham; and Duckworth Roger James "Machine editing of freehand work", UK Patent Application GB A, June 29, 1988 Integrated transparent touchscreen panel (transparent position-sensing membrane) and flat LCD display, with menu-driven user interface for handwriting/text recognition and sketch recognition/editing. [HillRD87a] Hill, Ralph D. "Supporting concurrency, Communication, and Synchronization in human-computer interaction - The Sassafras UIMS", ACM Trans. on Graphics Vol 5 No 3, July 1986, pp UIMS user-interface development system: not single-input stream or single application focus, instead Local Event Broadcast Method LEBM and Event-Response Language ERL for defining UI for multiple input devices, which may also be used for distinct tasks. Simultaneous use of mouse and touch-sensitive tablet for two-handed paint. Bimanual/multi-touch for two pointers? [Hilton88] Hilton, Colin Sefton "Method and Apparatus for capturing information in drawing or writing", European Patent , January 18, 1988 Signature verification tablet using optical sensor: refers specifically to recognizing characters (position) on paper which has different shading or color shading. Optical digitizer with specially-printed paper and styluses which reads shading of two colors in X and Y gradient, or shades of grey with different lighting sources. Compare with Sekendur. [Himelstein87a] Himelstein, Carol Sue and Wang, John Shihyan "Perspective projection", European Patent , May 6, 1987, assigned to IBM Corporation. User interface in 3-dimensional drawing: computer computes the 3rd depth dimension by user specifying a vanishing point [Himelstein87b] Himelstein, C.S. and Wang, J.S. "Editing method for interactive drawing", United States Patent 4,683,468, July 28, 1987, assigned to IBM Corporation. User interface for sketch input: match pointing position to vertices and endpoints of objects: editing subobjects within objects Identify objects by endpoints of writing motion [Himelstein87c] Himelstein, C.S. and Wang, J.S. "Editing method for interactive drawing", United States Patent 4,686,522, August 11, 1987, assigned to IBM Corporation. Identify objects by endpoints of writing motion [HiroakiN88a] Hiroaki, Negishi; Masaaki, Daimon; and Hiromi, Iwakura "Method for Displaying Window in Windowing System", Japanese Patent JPH (A), October 25, 1988 (Machine translation to English) Translucent/semi-transparent windows so that user can see objects underneath. Figure 6: transparent on-screen keyboard window. [HoltCM87a] Holt, C.M., Stewart, A., Clint, M., and Perrott, R.H. "An Improved Parallel Thinning Algorithm", CACM, Vol 30 No 2, February, 1987, pp Skeletonization/line-thinning on OCR, using 8 compass directions Line thinning with special provision for stair-case elimination [HopkinsD87a] Hopkins, Don "Directional Selection is as Easy as Pie Menus!", 4th Usenix Computer Graphics Workshop, March 30, 1987 (abstract only) Abstract of talk for Usenix on pie menus / marking menus. Menus can be organized semantically: counterpart operations in opposite radial directions, etc. [Huang87] Huang, J.S. and Chung, M.-L. "Separating Similar Complex Chinese Characters by Walsh Transform", Pattern Recognition, Vol 20 No 4, pp , 1987 O.C.R. for typed Chinese characters, first separation is by zoned template matching method, then by Walsh 8:26:29 PM]

149 ( ) History of Pen and Gesture Computing transform Says 5401 Chinese characters commonly used [HueckingEE87a] Huecking, Ernst E. and Klemmt, Harl-Heinz "Electronic data input keyboard comprising keys provided with conductive contacts", United States Patent 4,709,228, November 24, 1987 Matrix (row and column) keyboard allowing N-key roll-over / multiple key rollover for any number of keys. Row conductors are scanned sequentially by applying a voltage signal, column conductors receive scanning pulse of each row, so that every key can be read regardless of how many keys are depressed. Diode on each key switch prevents phantom key presses from multiple keys. Compare with Kaplow multi-touch, capacitive matrix keyboards and touch sensors. [IBM87a] IBM Corporation "IBM Scientists Demonstrate Personal Computer with Advanced Speech Recognition Capability", IBM Press Release, April 7, 1986 Talkwriter speech recognition, claims 95 percent accuracy, system must be trained to individual speaker's phonemes [IBM87b] IBM Corporation "Three-Axis Touch-Sensitive Pad", IBM Technical Disclosure Bulletin Vol 29 No 8 January 1987, pp, (partial copy) switch under touchpad/touchscreen so that user can "click" mouse button by pressing touchpad down. Alternative is force-sensitive third layer (implies two-layer touchscreen similar to elographics?) with higher activation force. Compare with MicroTouch mousepad/unmouse? [InfoWorld87a] Walkenbach, John "Product Comparison: Desktop organizers", InfoWorld, April 29, 1987, pp Comparison of products including Borland SideKick 1.5 with phone dialer: also Desktalk from Advance Logic Systems, Homebase 2.5 from Brown Bag Software, HQ 1.02 from TEK Microsystems, Maestro 1.0 from Spectre Software, Metro 1.0 from Lotus Development Corp., PC-Desk from Software Studios, Polywindows Desk Plus 1.2 from Polytron Corp., Pop-Up Desk Set Plus from Popular Programs Phone dialer from screen on Borland SideKick -- perhaps others also. [IngrahamRD88a] Ingraham, Ronald D. "Touch Control Switch Circuit", United States Patent 4,731,548, March 15, 1988 Capacitive sensing circuit for touch-only switch using voltage divider and phase detector circuit [IngrahamRD88b] Ingraham, Ronald D. "DC Touch Control Switch Circuit", United States Patent 4,758,735, July 19, 1988 Capacitive sensing circuit for touch-only switch using voltage divider and phase detector circuit [IshinoH88a] Ishino, Hiroaki "Optical Character Reader", United States Patent 4,751,743, June 14, 1988, assigned to Sumitomo Electric Industries, Limited, Osaka, Japan. Detecting edge of form and proper paper alignment for a hand-held optical scanner for OCR [KableRG87a] Kable, Robert G. "Position Responsive Apparatus, System and Method having Electrographic Application", United States Patent 4,678,869, June 7, 1987 Electrostatic grid digitizer tablet with grid of correction tables to transform detected coordinates to correct coordinates. Transparent conductors in two layers of grid thinner at crossing points to reduce electrostatic/capacitive coupling. [KableRG87b] Kable, Robert G. "Stylus for position responsive apparatus having electrographic application", United States Patent 4,695,680, September 22, 1987 Shielded stylus for electrostatic grid digitizer, stylus is the receiver [Kahan87] Kahan, S., Pavlidis, Theo, and Baird, H.S. "On the Recognition of Printed Characters of Any Font and Size", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol PAMI-9 No 2, March 1987, pp Confusion cases disambiguated through contour analysis on OCR Ligatures / merged characters broken and re-classified, then layout and linguistic context processing Many OCR systems fail accuracy if document is slightly tilted Error rate for manual typing / Human OCR is about 3 per page Practical OCR performance must be 99.9% accuracy, no substitution errors, just rejection Robustness is more important than accuracy (?) Lists many different kinds of character blobs/joins for OCR Linguistic context by certain very simple context heuristics, not grammar and punctuation rules Use UNIX spell dictionary for context manipulation in OCR Use of spelling dictionary does not help much correcting mis-recognition errors Test of OCR system, but omitted small punctuation, disconnected characters, ligatures, and special characters: confuseable errors not counted (!) Bayesian classifier assumes statistically independent features, which is not really true Performance of current OCR recognition system are glaring engineering problems, not science Need more theory of human reading ability and automatic design (?) of recognition classifiers For OCR, ligatures are usually accidental serif joins or double-o joins [KahnRE88a] Kahn, Robert E. and Cert, Vinton G. "The Digital Library Project. Vol. 1: The World of Knowbots (draft): An Open Architecture for a Digital Library System and A Plan for Its Development", Corporation for National Research Initiatives NRI, March 1988 Concept study for on-line digital library system DLS (as a national infrastructure): Indexing, registration of users, accounting/billing. "Knowbot" server would track source (author), copyright, ISBN / document ID, publisher, royalty compensations. 8:26:29 PM]

150 ( ) History of Pen and Gesture Computing [Kamal88] Kamel, Khaled and Iman, Ibrahim "A computerized transcription system for cursive shorthand writing", Proceedings of IEEE SouthEastCon, Knoxville, Tennessee, pp , April 1988 Cited by Goldberg for Unistroke recognition [Kankaanpaa88] Kankaanpaa, A. "FIDS - A Flat-Panel Interactive Display System", Laboratory of Information Processing Science, Helsinki University of Technology, Espoo 15, Finland, February 1987: IEEE Computer Graphics and Applications, Vol 8 No 3, March 1988, pp Electronic ink paper sent to me for review. Proof-reader's symbols in 1988 paper? simulation of recognition using a keyboard. User interface: electronic ink with gesture/special symbols for editing word-processing text with handwriting. Discusses "orthogonality" of gesture/command handwriting recognition symbols. Resistive tablet Touch-pen from Sun-Flex with two rings (?) as touch switch / tip switch. [Kaplan88] Kaplan, Samuel J., King, Jonathan J., and Sagalowicz, Daniel "Knowledge based processor for application programs using conventional data processing capabilities", United States Patent 4,783,752, November 8, 1988, assigned to Teknowledge, Incorporated, Palo Alto, California. Interface between artificial intelligence rule-based symbolic reasoning software and applications programs [Kashioka88] Kashioka, S., Shima, Y., Miyatake, T., and Ejiri, M. "Method for Producing a Standard Pattern for Pattern Matching", United States Patent 4,783,831, November 8, 1988, assigned to Hitachi, Limited, Tokyo, Japan. Used for machine vision, locating parts [KayA87a] Kay, Alan "Doing With Images Makes Symbols - Communicating with Computers", Video: Distinguished Lecture Series, Apple Computer, 1987 Video presentation on Dynabook, includes demonstration from 1960's (?) of gesture input in a drawing program using handwriting recognition of capital letters, GRAIL system, Sketchpad [KerrickDD88a] Kerrick, D.D. and Bovik, A.C. "Microprocessor-based Recognition of Handprinted Characters from a Tablet Input", Pattern Recognition, Vol 21 No 5, 1988, pp D.C.R. system in 32Kbytes of combined RAM/ROM: features are endpoints, corners, joins, angle, curvature, aspect ratio, relative positions D.C.R. system: three levels: low-level features and binary decision tree, binary tree for absence of features, high-level evaluation specific to remaining candidate characters D.C.R. system: retrace only allowed for certain lower-case characters: a, g, h, m, n, q, r, u, y Corner extraction using Freeman77 Stroke/character segmentation using possible fitting of strokes into a recognized character (bad idea: Easy to create lots of exception cases) This is NOT a robust system. There are too many ways to fool it -- and the authors don't cite much existing work, either. Uses HPR (Handprint Recognition) instead of D.C.R. or On-line character recognition Word segmentation based on overlap of 40% or more [KimJ87a] Kim, J. "Gesture Recognition by Feature Analysis", IBM Research Report RC (Log #56035) 1/28/87, IBM T.J. Watson Research Center, PO Box 704, Yorktown Heights, NY It takes more time to write a gesture than a character Asserts character recognition methods will not work for gestures Could not deal with "vocabulary explosion" by including rotational, inversion variants for gestures Recognizing all variants of a letter takes too much time and space (!) Gesture variation effects: non-linear scaling, rotation, mirror-image inversion, reverse direction Usually, a gesture by itself does not make much sense: compound phrases only Gesture recognition (at IBM) using twelve-direction chain codes [KimJ88a] Kim, Joonki "On-line gesture recognition by feature analysis", Proceedings of Vision Interface '88, Edmonton, June 6-10, 1988, pp Gesture recognition "differs from" handwriting recognition. Gesture recognition by direction changes (compare to Buxton SSSP reference), thus less subject to differences in rotational, mirror-image, reverse direction, size orientation for cut, pigtail, circle, other gestures. Gestures not independent of application, separate grammatical module for each application operation. Example application is Lotus spreadsheet. Gesture / direct manipulation interface to a spreadsheet program (Lotus 123), part of PLI paper-link interface projects at IBM. First stage of recognizer uses chain codes, second stage is custom-programmed for the gesture shapes. [Kimpan87] Kimpan, C., Itoh, A., and Kawanishi, K. "Fine classification of printed Thai character recognition using the Karhunen-Loe`ve expansion", IEE Proceedings, Vol 134 Pt E No 5, September 1987, pp Handwriting recognition for Thai: features are Karhuen-Loe`ve with maximum eigenvalue Features for Thai: Freeman chain codes, concavities and convexities, similar arc pairs: not conclusive results: 100% on training set, "lower" on test data (but not given) Cites other work on Korean, Arabic, Thai handwriting recognition [Kinoshita88] Kinoshita, Akiyoshi "Character Input System", United States Patent 4,724,423, February 9, 1988 Simple zone-type handwriting character recognition reading the strokes on the calculator keyboard [KishiH88a] Kishi, Hajimu and Tanaka, Kunio "Graphic display method for displaying a perspective view of an object on a CRT", United States Patent 4,754,269, June 28, :26:29 PM]

151 ( ) History of Pen and Gesture Computing Matrix algorithm for displaying an image of an object in a 3D projection on a display. Perspective view includes rotation. Compare with CRC Mathematical tables? [KobayashiK88] Kobayashi, K., Sakurai, A., and Sato, G. "Character Recognition Method and System Capable of Recognizing Slant Characters", United States Patent 4,757,551, July 12, 1988, assigned to Ricoh Company, Limited, Tokyo, Japan. Translating scanner input into direction codes: line-thinning. Recognition by histogram of contour direction/chain codes on OCR handwritten numbers. Asserts that slant/tilt varies among handwriting by different people, but consistent for one user [Kondo87] Kondo, Shozo "Some Considerations on Stroke-Structure of Characters Using a Model of Handwriting Process", Proceedings of the Third International Symposium on Handwriting and Computer Applications, Ecole Polytechnique de Montreal, Canada, 1987, pp [KramerSM88a] Kramer, Steven M. "On Incorporating Access Control Lists into the UNIX Operating System", Proc. USENIX UNIX Security Workshop, Aug , 1988, pp Report on vendor proposals for ACLs access control lists on UNIX, in response to DoD "Orange Book" (see DoD85). Cautionary essay on rush to implement vs. consideration of impacts on rest of operating systems, especially DAC discretionary access control (permissions/rights set by user) vs. mandatory access control (permissions set by system on user). Multiple ways to combine ACLs with traditional UNIX OGO (Owner, Group, Superusr) definitions. Wildcard syntax in ACLs. Denial of rights ("deny" permission) complicates things more. Extended permissions: examples of ownership (right to set permissions), extend (make file larger), truncate file, delete. [KreinPT88a] Krein, Philip T. and Meadows, R. David "The Electroquasistatics of the Capacitive Touch Panel", IEEE Conf. Proc CH2565-0/88/ Seems to be same as KreinPT90a: Indium Tin Oxide ITO resistive film touchscreen errors [KruegerMW88a] Krueger, Myron W.; Hinrichsen, Katrin; Gionfridd, Tom; and Sonnanburg, Joan "Videoplace '88", Studio in the Museum of Natural History, Vernon CT. June 10, 1988 Videoplace: video of exhibit using optical (TV image) detection of multiple fingertips (multi-touch), singletouch and multi-hand gestures for manipulating graphical objects and animations. [KurlanderD88a] Kurlander, David and Bier, Eric A. "Graphical Search and Replace", ACM Computer Graphics, Vol 22 No 4, August 1988, Scriptable/pattern-based technique for bulk editing of graphical images, by specifying a transformation of graphical objects whose properties, color, shape, etc. match a pattern or example. Scripting referred to as mouse-click macros. [KurlanderD88b] Kurlander, David and Feiner, Steven "Editable Graphical Histories", IEEE 1988 Workshop on Visual Languages, Pittsburgh, PA, October pp Undo/redo history lists for graphical input, modified to allow browsing of and re-execution of previous commands, taking into account different spatial state of graphical program. Shows Before/After previous of each operation that can be re-applied. [Kurta87a] Kurta Corporation "IS/PenMouse", Kurta Corporation, 1987 Technical specification on Kurta IS/PenMouse digitizer tablet, includes examples of binary format on serial port. Stylus or puck, each with additional buttons (on side of stylus, etc.) [Kurta88a] Kurta Corporation "Kurta out-does the competition with Architecture Templates", Kurta Corporation, 1988 On-table menus/command templates for VersaCAD, AutoCAD, electrical symbols, other PC/DOS CAD applications. Compare with similar offerings from CIC/ Pencept: this without handwriting recognition or gestures. [Kurta88b] Kurta Corporation "Kurta out-smarts the mouse", Kurta Corporation, 1988 On-tablet menus/command strips, described as additional user-programmable Function Keys for use with Apple MacroMaker for keyboard shortcuts. Cordless stylus to 12-button cursor (puck). [Kurtzberg87] Kurtzberg, J.M. "Feature Analysis for Symbol Recognition by Elastic Matching", IBM Journal of Research and Development, Vol. 31 (1) (1987) Combines feature analysis with elastic matching to filter out unlikely prototypes: pruning features for preprocessing recognition a number of strokes, points per stroke, relative heights of endpoints / maxima of strokes [KustanovichY87a] Kustanovich, Yosef "Surface-area pressure transducer and line-selection circuit for use therewith", United States Patent 4,644,801, February 24, 1987 Grid of capacitive pressure sensors with compressible material between the plates of the capacitors. Each capacitor can be measured individually. Used for biomedical instrumentation (pressure profile of foot) [Kuzunuki87] Kuzunuki, Soshiro, Shojima, HHiroshi, Yokoyama, Takanori, Fukunaga, Yashushi, and Hirasawa, Kotaro "Method for Designating a Recognition Mode in a Hand-Written Character/Graphic Recognizer", United States Patent 4,680,804, August 14, 1987, assigned to Hitachi, Limited, Tokyo, Japan. 8:26:29 PM]

152 ( ) History of Pen and Gesture Computing Projections of strokes on to X and Y axes for recognition: mode setting: strokes larger than a certain size are annotation, not characters. Simple size context to distinguish characters and drawing marks [KwokPCK88a] Kwok, P.C.K. "A Thinning Algorithm by Contour Generation", CACM, Vol 31 No 11, November 1988, pp Skeletonization/line-thinning with grey-scale information [LaLondeW88a] LaLonde, Wilf and Pugh, John "Smalltalk: Graphics Through the Looking Glass", JOOP, August 1988, pp Implementation of a magnifying-glass tool (with transparent window) in Smalltalk: "white hole" (transparent window section) with a border to highlight. [LakinF87a] Lakin, Fred "Visual Grammars for Visual Languages", Proc. AAAI-87, pp Two-dimensional visual languages using spatial parsing (e.g. mathematical expressions), shows handwritten input using tablet digitizer. Contains additional references back to 1971, for visual parsing of sketches of houses, etc. [LapeyreJM87a] Lapeyre, James M. "Electro-Optical Position Determining System", United States Patent 4,688,833, August 25, 1987 Optical digitizer/tablet for large area, two optical sensors on the edges of the tablet with triangulate to a light LED on the stylus [LeeS88a] Lee, Seongwhan, and Kim, Jin H. "Automatic verification of seal imprints using attributed stroke graph matching", Proceedings of the International Conference on Computer Processing of Chinese and Oriental Languages, 1988, pp [LeeS88b] Lee, Seongwhan, and Kim, Jin H. "Recognizing hand-drawn symbols in engineering drawings", Artificial Intelligence in Engineering: Robotics and Processes", Computational Mechanics, Southampton, 1988, pp [Leung87] Leung, C.H., Cheung, Y.S., and Wong, Y.L. "A Knowledge-Based Stroke-Matching Method for Chinese Character Recognition", IEEE Trans. Systems, Man, and Cybernetic, Vol SMC-17 No 6, November 1987, pp Relaxation method for distance from feature-space template with reduced computational load for Chinese recognition Minimum-distance template matching in feature space Trained/adaptive system, but with specific knowledge of writing Chinese characters, too Tested on 240 Chinese characters only, notes that extension to 4000 will add problematically to compute time User training sample collection (for only 240 Chinese characters!) is admitted to be tedious [Licklider88] Licklider, J.C.R. "Interview with J.C.R. Licklider, Conducted by William Aspray and Authur Norberg", 28 October, Charles Babbage Institute, Center for the History of Information Processing, University of Minnesota, Minneapolis. Available at Mentions Herb Teager's early work on a capacitive (?) coupling digitizer tablet, and the alternative project of a mouse: Page 20, with mouse and light-pen. [Linus87a] Linus Technologies "LINUS (tm) Write-Top User's Guide", Linus Technologies Inc, 1889 Preston White Dr., Reston, VA Shows writing/editing with handwriting recognition on a single surface display/tablet digitizer [Linus87b] Linus Technologies "Linus Write-Top / TRW Write-Top: Linus 1000", Posting at oldcomputers.net, fetched 2014 Touchscreen laptop computer, show calibration and application screens [Linus88a] Linus Technologies "Linus plans tiered Write-Top channel strategy", Product announcement for Write- Top handwriting recognition product, Computer and Software News, June 13, 1988 Product announcement, portable handwriting recognition product for Linus Technologies: Write-Top. Additional materials in file: Ralph Sklarew and Robert Nadeau, George Mason University: original name was "Techbook" [Linus88b] Linus Technologies "Equipment/Device: Linus Technologies Write-Top, ca. 1988", Linus Technologies (physical device) Early portable pen-computing, predecessor to PenPoint from GO, Microsoft PenWindows and Tablet PC. Touch-screen digitizer connected to display unit. (Stylus missing). [Linus88c] Linus Technologies "Linus Just-Write and Just-Write II Software Reference Manual", Linus Technologies, Inc, 1988 Handwriting recognition text-editing application. Desktop application shows text listing of files (similar to File Commander) on DOS operation system, for executing an opening files. Also has virtual keyboard. [Linus88d] Linus Technologies "Portable Computer Runs on Handwriting", Insight Magazine, August Linus Write-Top portable computer: Handwriting recognition directly on touchscreen, healthcare/nursing applications [LittlePD88a] Little, Paul D. "Audio/telephone communication system for verbally handicapped", United States Patent 4,785,420, November 15, 1988 On-tablet menus as talking UI for verbally handicapped using a touch digitizer: Chalkboard from Powerpad. 8:26:29 PM]

153 ( ) History of Pen and Gesture Computing Note that Powerpad (later models) accepted multi-touch input, resolution 0.1 inches. [LiuzzoJG87a] Liuzzo, James G. and Proettea, William J. "Keyboard with arrays of function keys", United States Patent 4,698,618, October 6, 1987 Physical keyboard with function keys near numeric row. Key to adjust display brightness. [LuP87] Lu, P.-Y. "professional resume", for work in on-line handwriting recognition Resume for former employee of Communications Intelligence Corporation. Device Driver and Mouse Emulator for digitizing tablet [Lukis87] Lukis, L.J. "Method and Apparatus for Correcting X-Y Position Measurements", United States Patent 4,679,241, July 7, 1987, assigned to Moore Business Forms, Incorporated, Grand Island, New York. Resistive-sheet digitizer tablet with correction table calibration Several references to British patents for portable handwriting recognition dynabook: diagrams show Moore Business Forms handwriting product (portable) [MaarseFJ88a] Maarse, F.J., Janssen, H.J.J., and Dexel, F. "A special pen for an XY-tablet", preliminary manuscript, University of Nijmegen, Department of Experimental Psychology, Montessorilaan 3, PO Box 9104, 6500 HE Nijmegen, The Netherlands, Published 1988 in "Human-Computer Interaction: Psychonomic Aspects", Springer Verlag, pp Measuring pen stylus tilt angle using peak levels of signal picked up in electromagnetic tablet: digitizer which reports stylus angle [Machart88] Machart, Beverly H., and Wang, John S. "Freehand Drawing Containing Invisible Lines", United States Patent 4,757,549, July 12, 1988, assigned to International Business Machines Corporation, Armonk, New York. Stroke segmentation in sketch recognition: group strokes together in a signature, or any free-hand writing or scribble, if they are close enough together Compares stroke starting and ending positions with previous storage ending to detect termination of a signature [MaedaK87] Maeda, Kenichi, and Nitta, Tsuneo "Pattern Recognition Apparatus and Method for Making Same", United States Patent 4,651,289, March 17, 1987, assigned to Tokyo Shibaura Denki Kabushiki Kaisha, Kawasaki, Japan. After training, a speech/handwriting recognition system stores deltas from previously learning patterns on misrecognition, not entire pattern Reduce memory size for pattern dictionary by storing deltas from originals, not whole patterns [MakkuniR87a] Makkuni, R. "A gestural representation of the process of composing Chinese temples", IEEE Computer Graphics and Applications, Vol 7 No 12, December 1987, pp [Malzbender87] Malzbender, T. "Permuted Trace Ordering Allows Low-Cost, High-Resolution Graphics Input", Hewlette-Packard Journal, June 1987, pp 4-7 Description of Hewlett-Packard's digitizer electrostatic tablet circuitry. By permuting traces in an X/Y grid, and reporting the numbers of which traces which detected a signal in decreasing strength, able to interpolate very high accuracy without relying on absolute measurements of signal strength. Amplifier built into stylus. HP 45911A digitizing tablet. [Mandler87] Mandler, Eberhard "Advanced Preprocessing Technique for On-Line Script Recognition of Nonconnected Symbols", Proc. 3rd Int. Symp. on Handwriting and Computer Applications, pp , July 1987 Cited by Marlin Eller, Microsoft Pen Computing group [ManessWL88a] Mannes, William L.; Golden, Robert F.; Benjamin, Michael H.; and Podoloff, Robert M. "Contact Sensor for Measuring Dental Occlusion", United States Patent 4,734,034, March 29, 1988 Thin pressure sensor (like a touchpad), used for checking dental occlusion. X/Y grid of conductors, separation layer between. Separation layer is very thing nylon mesh, or talcum powder. Under pressure, more of grid conductors come into contact. Resistance across separation layer goes abruptly to very low at a pressure threshold. No explanation for "ghosting"? [MatthewsHG88] Matthews, Henry G.; Zalenski, Thomas; Barbetti, Jamie L; and Mletzko, Al "Menu for a Charge Ratio Digitizer", United States Patent 4,788,386, November 29, 1988 Summagraphics charge-ratio tablet, with tapered traces and varying-width traces on a PC board [MatzkeKE88a] Matzke, Karl E. and Schick, Paul W. "Touch Activated Control Method and Apparatus", United States Patent 4,736,191, April 8, 1988 Circular capacitive touchpad (for keyboard), radial plates: which plate touch controls direction of movement of cursor, pressure (as inferred from change in capacitance) determines velocity of cursor. Tap and dwell input on touchpad for control functions. Dwell/press-and-hold. [McArthurD87a] McArthur, David; Stasz, Cathleen; and Hotta, John Y. "Learning Problem-Solving Skills in Algebra", RAND Note N-2595-NSF, Prepared for The National Science Foundation, May 1987 (abstract only) see 1990 publication by same authors [McAvinneyP88a] McAvinney, Paul "Method and apparatus for isolating and manipulating graphic objects", United States Patent 4,746,770, May 24, 1988 Optical beam touchscreen device, multiple fingers. See McAvinney bio reference: states was first optical 8:26:29 PM]

154 ( ) History of Pen and Gesture Computing multi-touch device. [McCaskillRA88a] McCaskill, R.A., Wang, J.S., and Repass, J.T. "Printed document processing", United States Patent 4,739,314, April 19, 1988, assigned to IBM Corporation. User-interface: specifying measurements of printed document objects by pointing to them [MelBW88a] Mel, Barlett W.; Omohundro, Stephen M.; Robison, Arch D.; Skiena, Steven S.; Thearling, Kurt H.; Wolfram, Stephen; and Luke, Young T. "Tablet: The Personal computer of the Year 2000", Communications of the ACM, June 1988, pp ; also Report No UIUCDCS-R /UILU-ENG , U. Ill. at Urbana- Champaign, February 1988 Forerunner proposal to iphone/ipad? Winning proposal in Apple-sponsored competition to envision the computer of the future. Touchscreen / tablet (stylus for high resolution input) used as a mouse. Gesture for turning on is to touch two corners. Voice input/output (multi-touch? or just didn't know tablet technology?): no other gestures?. Built-in camera for scanning/digitizing documents, video camera , videoconferencing, GPS with complete internal map storage. Social apps for finding interesting people nearby. Interactive group gaming. [Microsoft87a] Microsoft Corporation "Microsoft Excel Arrays, functions and Macros", Microsoft Corporation, 1987 (hardcopy book) Description of Microsoft Excel spreadsheet program for Windows [Microsoft87b] Microsoft Corporation "Reference to Microsoft Word Word Processing Program Version 3.0 for the Apple Macintosh", Microsoft Corporation, 1987 (hardcopy manual) Word 3.0 Word processing for the Apple Macintosh - shows integration with Apple menus. Contains update for version 3.01 describing differences from Word 1.0 for the Macintosh. [Microsoft87c] Microsoft Corporation "Learning Microsoft Word: Word Processing Program Version 3.0 for the Apple Macintosh", Microsoft Corporation, 1987 (hardcopy book) Word 3.0 Word processing for the Apple Macintosh - shows integration with Apple menus. Contains update for version 3.01 describing differences from Word 1.0 for the Macintosh. [Microsoft88a] Microsoft Corporation "Descriptions of Expert and Standard Mouse Menus", support.microsoft.com, article / Q23837, approximate date 1988 Description of MENU.COM, *.MNU files, MENUMAKE.DOC, MENUREAD.ME. Mouse menus allow for support of mouse in DOS applications that are not "mouse-aware". MENU.COM uses static menu definitions, expert menus are *.COM files with specific knowledge of application. Compare with keyboard macros, and with recognition macros by PenCept and CIC for handwritten gestures. File contains additional references concerning mouse menus for Application That Do Not Use the Mouse, standard keyboard layouts Keyboard Styles fro IBM and IBM Compatibles, Using Microsoft MOuse Menu Software with MS-DOS 4.x-6.0 (MenuMaker for mouse menus not compatible with extended key codes) etc. [MiletzkiU87a] Miletzki, U; Doster, W.; Fogaroli, G.; Lobl, H; and Moulds, P. "Paper Interfaces for Office Systems", ESPIRIT '86, pp , 1987 Cited in FordDM91a: regarding gestures [Mishima88] Mishima, T., Kanasaki, M., Takatoo, M., and Ota, H. "Half-tone image recognition / Method and apparatus for recognizing pattern of halftone image", European Patent , October 26, 1988, assigned to Hitachi, Limited, Tokyo, Japan. Image processing: recognize halftone texture in a scanned image, to do OCR character recognition without binarization, so that contrast/brightness does not affect recognition [Mochinaga88] Mochinaga, N., Moto, T., and Ogata, Y. "Pen-type Character Recognition Apparatus", United States Patent 4,751,741, June 14, 1988, assigned to Casio Computer Company, Limited, Tokyo, Japan. Pen stylus which measures lateral pressure/force using strain gauges for handwriting recognition, instead of an X/Y digitizer : used for handwriting recognition / character recognition [Morishita87] Morishita, T., Ooura, M., and Ishii, Y. "A Kanji Recognition Method Detecting Incorrectness in Writing", Proceedings 1987 International Conference on Computer Processing of Chinese and Oriental Languages", pp Claims 99.7% recognition of Kanji characters "with errors", 99.9% for "correctly written" Chinese characters, used to check for correct writing style in Japanese, "Standard Writing" by the Japanese Council on the National Language [MullerMJU88a] Muller, Michael J. "Multifunctional Cursor for Direct Manipulation User Interfaces", Proc. CHI '88, pp Cursor showing state of current selected tool: suggests first selecting tool and then selecting object to apply it to (in the meantime, cursor indicates which tool is "active), rather then first select object and then select operation/command to apply to it. Update (pop-up?) menus/icons on mouse input? Mentions implementation for 1/2/3-button mouse, with single- and double-click actions. [MyersBA87a] Myers, Brad. A. "Creating Interaction Techniques by Demonstration", IEEE CG&G Computer Graphics and Application, September 1987, pp :26:29 PM]

155 ( ) History of Pen and Gesture Computing Peridot: GUI generator for direct-manipulation interfaces, action of user interface can be defined by defining default/initial parameter values (internal values) and running an example -- and then editing code. Supports two-handed bimanual (multi-touch) input by supporting multiple locator input devices. [MyersBA88a] Myers, Brad. A. "The Garnet User Interface Development Environment: A Proposal", Carnegie- Mellon University Technical Report CMU-CS , September 1988 In Garnet system papers: set of UIMS/UI design tools for direct manipulation GUIs. Common Lisp IDE, but objects use prototype inheritance, not classes. Any change in property of an object immediately updates display of object (e.g. color management, color properties, or uncovering a window) [NCSC87a] National Computer Security Center "Trusted Network Interpretation", NCSC-TG-005 Version 1, Library No. S228,526, 31 July 1987, Gallagher, Patrick R. Jr. Director. Available at csrc.nist.gov (2015) Extension of NCSC Trust System Evalution Criteria (Dod85) to trusted network systems and components. Communications integrity, physical security: specifics of encryption outside of scope. [NYTimes88a] Lewis, Peter "The Executive Computer -- Exotic Gizmos for Aiding Workers", November 20, 1988, New York Times ComDex: Private Eye head-mounted single eye display monitor, Reflection Technology Ink: Wang Freestyle combining text, handwriting (electronic ink notes), voice recording, electronic mail (via FAX) and filing. Mentions Scriptel tablets. [Nadler87] Nadler, Morton "private communication with Morton Nadler", August, 1987 This fellow was in handwriting character recognition in the 1960's, on ANSI handprint-standard committee as GE representative NADLer Inc., Blacksburg VA (circa 1986) [Nakamura87a] Nakamura, Shoichiru, and Kable, Robert "Electrographic System and Method", United States Patent 4,650,926, assigned to Scriptel Corporation, March 17, 1987 Correction grid matrix for digitizer: divide into small correction rectangles [NationalPhysicalLaboratory87] National Physical Laboratory NPL "Electronic Paper: The Electronic Paper Collaboration", National Physical Laboratory NPL, Division of Information Technology and computing, Tedding, Middlesex TW11 0LW, United Kingdom Scriptel digitizer and plasma display, with handwriting/cursive recognition. Commercial development by DataSystems UK. Says "Electronic Paper is nearly here" (sic). [Nestor87] Nestor, Inc. "NestorWriter Product Description", available from Nestor Incorporated, One Richmond Square, Providence RI Cited in --->"Computing with Neural Networks", High Technology, Vol 7 No 5, May 1987, pp References to Nestor's work on handwriting recognition product [Nestor88] Nestor, Inc "Press/correspondence file on Nestor, Inc.", (personal correspondence) References to Nestor's work on handwriting recognition products and technology using neural nets. Mike Buffer, Leon Cooper. [NielsenJ87a] Nielsen, Jakob "CHI+GI '87 Trip Report", Mentions demonstration of DataGlove, with a hand-gesture of opening the fingers of the hand to clear a graphic screen (swipe gesture?). Demo of Alternate Reality Kit with direct manipulation of simulated objects using a mouse. (See Alternate Reality Video) [NielsenJ88a] Nielsen, Jakob "CHI '88 Trip Report", Refers to "two-cursor" problem, but this has nothing to do with multi-touch multiple touch inputs on a digitizer or tablet, it is a cognitive problem of needing to point simultaneously to a command (e.g. "paste" icon) and a target (position in text). [Numonics88a] Numonics Incorporated "#2200 Digitizer Pad Technical Information", Available from Numonics Incorporated [Numonics88b] Numonics Incorporated "CADCommand product information", Available from Numonics Incorporated, Montgomeryville PA Pencept PenCAD and PenPad 320 sold under Numonics name: recognition macros / gestures, handwriting recognition [Numonics88c] Numonics Incorporated/Terminal Display Systems Limited "MAX - Multi-Axis Input Digitizer", "ZedPEN product literature", Terminal Display Systems, Lower Philips Road, Blackburn England BB1 5TH, axis digitizer (X,Y, theta/phi/rho), plus force/pressure sensor, plus "flick" accelerometer sensor [Numonics88d] Numonics Incorporated "Numonics Corporation covers computer graphics from architecture to zoology", Numonics Corporation, 1988 Numonics product line (pre-pencept), shows electromagnetic and electromechanical 2D digitizer tablets. Models 1210, 1220, 1240, , 1224CF electromechanical digitizers, Models 200, 2300, 2400 tablets, with 16-button cursor/puck, stylus. Electromechanical include firmware for computing areas. [OoiK87a] Ooi, K., Hidai, K., Kurosawa, Y., and Nakamura, Y. "Method and Device for Handwritten Letter Recognition", United States Patent 4,685,142, August 4, 1987, assigned to Kabushiki Kaisha Toshiba, Kawasaki, Japan. 8:26:29 PM]

156 ( ) History of Pen and Gesture Computing Kanji/Japanese recognition using standard strokes, in a standard stroke order Attempt to remove strokeorder properties from Kanji recognition using a hierarchical dictionary of standard strokes, and matching as you go [OritaM87a] Orita, M., Kobayashi, Y., and Kuboo, Y. "Tree-structure recognition dictionary preparation system", United States Patent 4,658,429, April 14,, 1987, assigned to Hitachi, Limited, Tokyo, Japan. Order tree nodes in a recognition dictionary of templates, so features with the most power are highest in the tree [OritaM87b] Orita, M., Mishima, T., and Kobayashi, Y. "System and Method for Preparing a Recognition Dictionary", United States Patent 4,682,365, July 21, 1987, assigned to Hitachi, Limited, Tokyo, Japan. Hierarchical tree-structure for pattern recognition decision tree: method to minimize search time by organizing tree (for Kanji recognition) Cites prior art (1975) on tree-structured pattern recognition dictionaries for Kanji/Japanese recognition [Ovonics88] Ovonic Imaging Systems OIS, Inc. "E-Z image and "OIS Telepad" product literature, 1988", 1896 Barrett Street, Troy, MI 48084, EZ-Image is a clear digitizer tablet, resistive sheet: electronic ink with integrated LCD display Telepad is an integrated digitizer/display unit Uses the term "electronic paper" (similar to "electronic ink") [PearsonG87a] Pearson, G. and Weiser, M. "Exploratory Evaluations of Two Versions of a Foot-Operated Cursor- Positioning Device in a Target-Selection Task", poster paper, CHI+GI 87 Conference, Boston, Massachusetts, April Authors' address: Heterogeneous Systems Lab, University of Maryland, College Park, Maryland. Foot-operated pointing device, similar to mouse [PenMouse88a] PenMouse Technologies "Equipment/Device: PenMouse device, ca. 1988", PenMouse, PenMouse Technologies (physical device) Roller-ball mouse in a pen-shaped housing, does not report absolute position. Not the same as the IS/PenMouse digitizer tablet from Kurta. [Pencept88a] Pencept "PenPad Digitizer and Intelligent Tablets -- Using Artificial Intelligence to Make Your Computer Faster and Easier to Use", Pencept division of Numonics, Inc., 39 Green Street, Waltham, Massachusetts 02154, 1988 Pencept PenPad 300, 310, 320 electromagnetic tablets, handwriting recognition, command macros (gestures) [PersonalWriter88a] Personal Writer Inc. "Personal Writer PW15 SL User Manual", Personal Writer Inc., 1801 Avenue of the Starts Suite 507, Los Agneles CA (hardcopy book) Handwriting recognition tablet for MacIntosh, unconnected/discrete letters -- no boxes, but characters must be widely spaced. Trainable recognition. On-tablet menus and tablets with drawing, recognition, command, and menu areas, macro definitions, mouse areas. Recognition makes use of personally defined text dictionaries. Keyboard emulation, mouse emulation with tablet. Remarkably similar functionality to PenCept PenPad and CIC HandWriter. Includes sheets for user to write 5 examples of each character/symbol shape to be recognized. [Phillips87] Phillips, M. "Several simple tests can help you choose the correct digitizer", Computer Technology Review, Vol VII No 1, January 1987 Mark Phillips / Ted Kuklinski tablet tests: includes tilt measurements [PierK88a] Pierk, Ken; Bier, Eric; and Stone, Maureen "An Introduction to Gargoyle: An Interactive Illustration Tool", Proc. Int'l. Conf. on Electronic Publishing, Document Manipulation and Typography, Nice France, April 1988, p Gargoyle graphical drawing editor: uses snap-dragging instead of snapping to alignment grid for precise positioning. Snap-dragging: objects may have "gravity", causing snap to a control point or to a surface/line if the cursor comes close enough. Refers to Cedar context-sensitive pop-up menus: compare to marking menus -- user can hold cursor over object, and long-press on any mouse button to get the menu -- pressand-hold? [Platshon88] Platshon, Mark "Acoustic touch technology adds a new input dimension", Computer Design, March 15, 1988, pp Technical description of Elographics digitizer technology, surface acoustic wave (SAW). SAW technology for force/pressure-sensitive "Z" on touch tablet digitizer. Refers to finger as a "pliable pointer", system can infer force from effect of increasing contact area: pressure/force gestures e.g. light touch for drag, firm/heavy touch for click/activate. [Pobgee88a] Pobgee, P.J. "Prototype System for Interactive Input of Cursive Information", National Physical Laboratory, Teddington (England), Division of Information Technology and Computing, Report No NPL-DITC- 125/88, 1988 (abstract only). NTIS citation index: refers to pointing and handwriting recognition combined in a GUI application. User interface and editing and graphics manipulation for handwriting recognition. [Pobgee88b] Pobgee, P.J. "EPT, a Dynamic Tutorial for Introducing New Users to Electronic Paper", National 8:26:29 PM]

157 ( ) History of Pen and Gesture Computing Physical Laboratory, Teddington (England), Division of Information Technology and Computing, Report No NPL- DITC-134/88, 1988 (abstract only). NTIS citation index Integrated tablet and display with sample handwriting applications [PotterR88a] Potter, R., Weldon, L., and Shneiderman, B. "Improving the accuracy of touch screens: An experimental evaluation of three strategies", Proceedings of CHI conference, Washington, D.C., May 15-19, Published by the Association for Computing Machinery, New York, 1988, pp Touch screens advantageous/desirable because do not consume workspace, no moving parts. Accuracy limitations due to visual parallax: target on touch-down, target on drag-to-touch; target on lift (to be patented) -- does not note that target-on-lift typically employed in mouse systems, albeit for different reasons. Mentions highlighting before lift, so that user can tell what would be selected if user lifted pen then. Mack89 digitizer user-interface hacks [RabinMO88a] Rabin, Michael O. and Tygar, J.D. "An Integrated Toolkit for Operating System Security", Harvard University Aiken Computation Laboratory, Technical Report TR-05-87R, August, 1988 Model for secure/trusted computer systems: indelible protections (attached access rights) as passive protections, sentinel (active program) protection. Fingerprinting (invariant condition checks?)_ at stages of kernel execution to detect race-condition insecurities in system calls. Cites to DOD85. [RaymondDR87a] Raymond, Darrell R. "Using Dispatchers to Control Process Structures", Univ. of Waterloo CompSci Research Report CS Blocking send/receive (read/write) system extended with "courier" intermediate process to permit nonblocking process control, without full buffering. Creating and management of courier by central dispatcher. Does not mention rights or protection enforcement by intercepted/mediated messages. [Reilly87] Reilly, Douglas, L., Scofield, Christopher, Elbaum, Charles, and Cooper, Leon, N. "Learning System Architectures Composed of Multiple Learning Modules", Proceedings of IEEE First Annual International Conference on Neural Networks, June, 1987, Vol II, pp Recognition training time is shorter if you train one module to each shape class, rather than many modules to share training, in a Nestor neural network. [Reilly88] Reilly, Douglas L, Scofield, Christopher, Gouin, Philip R., Rimey, Raymond, Collins, Edward A., and Ghosh, Sushmito "An Application of a Multiple Neural Network Learning System to Industrial Part Inspection", to be presented at ISA/88, Houston, Texas. Nestor recognition paper on industrial vision parts inspection [Rhyne87] Rhyne, J. "Dialogue Management for Gestural Interfaces", Computer Graphics, Vol 21 No 2, April 1987, pp Also IBM Research Report RC 12244, 1986 Temporal grouping/segmentation of written gestures more important than the temporal sequence [RodgersJL87a] Rodgers, James L. and Waterbury, Charles A. "Low power high resolution digitizing system with cordless pen-mouse", United States Patent 4,672,154, June 9, 1987 Capacitive/electrostatic digitizer with X/Y grid. Stylus is battery powered. Interpolation between grid lines. Stylus buttons sensed by changing transmitting frequency of pen slightly. [RosenbergJM87a] Rosenberg, Jerr M. "Dictionary of Computers, Information Processing and Telecommunications, Second Edition", Wiley, 1987 (hardcopy book) Technical dictionary of computer terms [RubinT88a] Rubin, Tony "User Interface Design for Computer systems", Ellis Horwood Books in Computer Science, U. of London, 1988 (hardcopy book) User interface guidelines concerning keyboard commands, error messages, text menus. Section on WIMP Interfaces: keyboard, mouse, joystick, trackball, touch sensitive screens, light pen, speech input / speech recognition. Oldest touchscreen stated as touch wire touch sensitive screens / TSS -- also mentions conductive grid: same thing? WIMP interface/ GUI examples taken from MacIntosh. [RubineD88a] Rubine, Dean and McAvinney, Paul "The Videoharp", Proc. ICMC 88, pp Electronic VideoHarp musical instrument using optical sensing of multiple fingers (multi-touch) and gestures: either harp-like (strum) or piano-like (projected keyboard). Optical sensing on two sides (either side) of transparent instrument, such that instrument can be played with both hands like a harp, or mounted horizontally like a piano. MIDI output. to a synthesizer. Position of finger along a ray detected by shadowing. File contains additional materials showing Video harp construction, frame, and use. See also McAvinneyP90a [Rympalski87] Rympalski, William P., Herstein, James S., Ritenour, Roger L. "Electronic Sketch Pad", United States Patent 4,639,720, assigned to Harris Corporation, Melbourne, Florida, January 27, 1987 Capacitive digitizer integrated with an LCD display: patent on "writing" directly on a display. conformable stylus tip allows for indirect pressure sensing. Mutual capacitance, forms image of objects on display, therefore also multi-touch? [SAC87a] Science Accessories Corporation "GP-7 Grafbar Mark II Operator's Manual", Science Accessories Corporation, 1987, 970 King's Highway West, Southport, Connecticut :26:29 PM]

158 ( ) History of Pen and Gesture Computing Sonic/acoustic digitizer tablet, notes that has slant range correction, resetable coordinate origin, physical diagrams of stylus/pen and cursor/puck. Notes that drafts affect performance, wind changes speed of sound (?). Spark gap as acoustic source. [SAC87b] Science Accessories Corporation "QUESTOR III Model GP-9 two-dimensional sonic digitizer -- large areas -- projected images", Science Accessories Corporation, 1987, 970 King's Highway West, Southport, Connecticut Sonic/acoustic digitizer tablet, 2D, multi-button puck or stylus, large format. Has single large transducer/sensor bar. Mentions 3D models with up to 9x9x9 foot volume. Special requirements can be addressed. [SAC88a] Science Accessories Corporation "Capture / Create / Analyze / Direct with sonic digitizers", Science Accessories Corporation, 1988, 970 King's Highway West, Southport, Connecticut Sonic/acoustic digitizer tablet, 3D, up to 9x9x9 foot volume. [SIGSOFT87a] SIGSOFT "Anti-Skid Brakes", quoted from October 1986 issue of Road and Track, A.C.M. SIGSOFT Software Engineering Notes, Vol 12 No 1, page 10, January, 1987 Unanticipated/intermittent input/event causes software system error [SIGSOFT87b] SIGSOFT "Cause of the Mysterious Bay Area Rapid Transit Power Outage Identified", quoted from July 8, 1987 San Francisco Chronicle, A.C.M. SIGSOFT Software Engineering Notes, Vol 12 No 4, page 4, January, 1987 Unanticipated/intermittent input/event causes software system error [SIGSOFT87c] SIGSOFT "Actual Stock Price Fails Sanity Check", contributed by Mark Brader, A.C.M. SIGSOFT Software Engineering Notes, Vol 12 No 4, page 4, October 1987 Unanticipated/intermittent input/event causes software system error [SIGSOFT87d] SIGSOFT "BBC Documentary Filming Causes Library of Congress Computer Crashes", contributed by Howard C. Berkowitz, A.C.M. SIGSOFT Software Engineering Notes, Vol 12 No 4, page 15, October 1987 Unanticipated/intermittent user input/event causes software system error [SIGSOFT88a] SIGSOFT "Runaway mouse problem in popular commercial WP program", contributed by Steven Jones, A.C.M. SIGSOFT Software Engineering Notes, Vol 13 No 4, page 9, October 1988 Press report of mouse/driver (?) but in Word 4.0, Microsoft mouse on PS/2 port generates runaway input (buffer bug?) [Saba87] SABA "Handscan product literature", Saba Technologies, Incorporated, 9300 S.W. Gemini Driver, Beaverton, Oregon, 97005, June 1987 Produce literature on hand-held OCR input scanner Acoustic/sonic 2-D tablet [Satoh88] Satoh, Koji "Method of Character Recognition", United States Patent 4,783,835, November 8, 1988, assigned to Sumitomo Electric Industries, Limited, Osaka Japan. Typed character recognition in a hand-held scanner using right-side and left-side profiles/chain codes, in addition to line features [Schoonard87] Schoonard, James W.; Gould, John D.; Bieber, Miriam; and Fusca, AAngie "A Behavioral Study of a Computer Hand Print Recognition System", IBM Research Report RC (Log #56157) 2/6/87, 1987, IBM T.J. Watson Research Center, PO Box 218, Yorktown Heights, NY Compared human recognition accuracy, both by original writer and by another subject, with IBM's machine recognition In adaptive system, adding poorly-formed (atypical) prototypes from handwriting to system is a problem, as accuracy goes down In adaptive system, accuracy goes down as prototypes are added beyond a certain point (so why add them? when to add them?) Handwriting interaction tested for four tasks: copy, compose, spreadsheet, delayed copy/transcription Human recognition accuracy measured as 88.5% to 96.1%, average 93.2% Writing speed for hand-printing is 0.99 characters/second 90 to prototypes in adaptive system for individual subject's handwriting Subjects consistently tended to underestimate recognition accuracy of IBM system, because errors in words and inability to correct were so annoying 96% recognition accuracy "not good enough" for commercial use of on-line character recognition [ScottWC87a] Scott, Walter C. "Signature Verification", United States Patent 4,701,960, October 20, 1987, assigned to Texas Instruments Incorporated, Dallas, Texas. Signature verification using matches at local extrema points(?) and something like elastic matching [Sejnowski87] Sejnowski, Terrance and Rosenberg, Charles "Parallel Networks that Learn to Pronounce English Text", Complex Systems, 1: , (1987) Hinton diagrams; training input consisted of phonetic transcriptions of English text, not actual spoken recording. Cited by Marlin Eller, Microsoft Pen Computing group [Sekita88] Sekita, I., Toraichi, K., Mori, R., Yamamoto, K, and Yamada, H "Feature extraction of hand-written Japanese characters by spline functions for relaxation", Pattern Recognition, Vol 21 No 1, January 1988, pp 9-17 Cited in Yu90: for line thinning with context by chain codes [SemioticsDesigners87a] Semiotics Designers "Handwriting recognition product outline: Correspondence file", 2980 Salem Drive, Santa Clara, California 95051, :26:29 PM]

159 ( ) History of Pen and Gesture Computing Carol Anne Ogdin, formerly Mr. Ogdin Lists of sentences using all 26 letters of alphabet in various combinations, for collecting writing samples Sample list of sentences using all 26 letters of the alphabet -- source unknown [SherrS88a] Sherr, Sol (Ed.) "Input Devices", Academic Press Inc., 1988, ISBN (electronic book) Chapter 1, "Introduction to Input Devices", Caswell, N.S. Chapter 2, "Human Factors Considerations in the Design and Selection of Computer Input Devices", Arnaut, Lynn Y. and Greenstein, Joel S. Chapter 3, "Keyboards", Greenstein, Joel S. and Muto, William H. Chapter 4, "Digitizer and Input Tablets", Davies, Thomas E., Mathews, H. Gerard, and Smith, Paul D. Chapter 5, "Mice", Goy, Carl Chapter 6, "Trackballs and Joysticks", Doran, David Chapter 7, "Voice Input Systems", Viglione, Sam S. See also citations for each chapter. Windows input managers / window management, virtual devices. [Shojima87] Shojima, H., Kuzunuki, J. S., and Hirasawa, K. "On-line pattern recognition for hand-written shapes", United States Patent 4,653,107, March 24, 1987, assigned to Hitachi, Limited, Tokyo, Japan. Re-order strokes, feature is quantized curvature, try deleting endsegments or combining sequential strokes On-line handwriting recognition using a dictionary of templates: used in a document preparation userinterface? [ShojimaH88a] Shojima, H., Mifune, T., Mori, J. and Kuzunuki, S. "Method and Apparatus for On-line Recognizing Handwritten Patterns", United States Patent 4,718,103, January 5, 1988, assigned to Hitachi, Limited, Tokyo, Japan. Cites DP Matching Process to Character Recognition of Nikkei Electronics, Angular vector chain code sequences: matching successive angle differences for line segments in handwriting character recognition; Unanticipated/intermittent input/event causes software system error [SilversteinA88a] Silverstein, Alan; McMahon, Bill; and Nuss, Greg "Adding Access Control Lists to Unix; or, How to Stop Worrying About All that Other Security Stuff and Love UNIX DACLMs", Hewlett-Packard Company, March 12, 1988 (submitted to /usr/group?) cited in various patents: Discussion of practical aspects of different ways of implementing Access Control Lists ACLs in Unix, combining with traditional access bits. Cites to earlier work at HP since January 1987, an to the National Computer Security Center. Trusted computer base TCB of computer system. Discretionary Access control DAC is right to pass access rights/permission to others. Unix chmod bits as a form of DAC. [Sinha88] Sinha, R.M.K. and Prasada, B. "Visual Text Recognition Through Contextual Processing", Pattern Recognition, Vol 21 No 5, 1988, pp Spelling context dictionary where words outside the dictionary are handled with a Viterbi algorithm A priori context (i.e. statistical values for letter pair digrams) vs a posteriori context (checking results against a spelling dictionary) Contains quick overview of everybody's work on spelling or statistical probability context correction Transient dictionary for context: words not in regular Uses weighted confusion matrix of touching/rejected characters for context correction (segmentation) Refers to ambiguous upper/lower-case confusion pairs, other than the obvious In OCR, attempt to fix touching characters by a separate segmentation algorithm on rejected characters common "omitted" words can be used to correct the rest dictionary may occur frequently in document, so most [SklarewR88a] Sklarew, Ralph "Handwritten keyboard-less entry computer system", European Patent a, 1 Janurary 27, 1988, assigned to Linus Technologies, inc. Broad patent on Linus' keyboardless computer product, using handwriting recognition Gives many example screens for Linus/Grid's user interface, showing pop-up windows/rectangles for handwriting entry Gives figures for handwriting digitizer tablet performance: 0.015"/0.005", 150 points/second Linus/Grid userinterface on editing functions/symbols, appear similar to gesture-based user interface [Skylight88] Skylight Software, Incorporated "Handprints 2.0 product information", 323 Andover Street, Wilmington, Massachusetts Yuri Litvin's small on-line handwriting recognition software product: runs in 13Kbytes (!) [SmithR87a] Smith, Randall "Experiences with the Alternate Reality Kit: An Example of the Tension between Literalism and Magic", IEEE Computer Graphics, Sep vol 7 no 9, pp See also video of Alternate Reality Kit ARK: mouse-hand can manipulate literal ("realistic" but 2-D) objects with momentum, bouncing, etc. Other "magical" features such as buttons, changes to simulated gravity, etc. Suggests that multiple mouse buttons (by extension, mouse gestures?) are hard to learn because they are unnatural. [SmithRW87] Smith, R.W. "Computer processing of line images: A survey", Pattern Recognition, Vol 20, 1987, pp 7-15 Cited in Taxt90 [SpeechSystems87] Speech Systems, Incorporated "Phonetic Engine (R) product description", Oxnard Street, Tarzana, California 91356, 1987 Product literature on phonetic speech recognition system with large dictionary. Phonetic Decoder (TM), 8:26:29 PM]

160 ( ) History of Pen and Gesture Computing Phonetic Profiler (TM), DS100 application development system, [Srihari87a] Srihari, Sagur N. and Hull, Jonathan J. "System to Achieve Automatic Recognition of Linguistic Strings", United States Patent 4,654,875, assigned to Research Foundation of State University of New York, Albany, N.Y., March 31, 1987 Automatic language (actually, character) recognition by converting words to bit strings, and comparing resulting strings to the best match of similar bit strings: stated as applicable to OCR [Srihari87b] Srihari, Sagur N. and Bozinovic, Radmilo M. "A Multi-Level Perception Approach to Reading Cursive Script", Artificial Intelligence, Vol. 33 No 2, p , 1987 Cursive script recognition [StefikM87a] Stefik, Mark; Foster, Gregg; Bobrow, Daniel G.; Kahn, Kenneth; Lanning, Stan; and Suchman, Lucy "Beyond the chalkboard Computer: Support for collaboration and problem solving in meetings", CACM, Vol 30 No 1, January 1987, pp Cognoter/Colab: Xerox/PARC whiteboard/teleconferencing system supporting WYSIWIS what you see is what I see, users operated concurrently/in parallel on shared objects. Mouse or pen graphical input. Cites to Wang Freestyle. [StevensMJ88a] Stevens, M.J. "A Text Editor Driven by Hand-Drawn Symbols", NPL Report DITC 124/88, 1988 Cited in FordDM91a regarding gestures [StollC88a] Stoll, Clifford "The Cuckoo's Egg: Tracking a Spy Through the Maze of Computer Espionage", Pocket Books, ISBM Discrepancy of 75 cents in a billing program for internet usage led SysAdmin to first public uncovering of an attacker/hacker group is fraudulent [Suen88] Suen, C.Y. (chair) "Future Challenges in Handwriting and Computer Applications", panel discussion, 3rd International Symposium on Handwriting and Computer Applications, Montreal, May 29, Panelists: R. Plamondon, C. Tappert, A.J.W.M. Thomassen, J.R. Ward, K. Yamamoto. Summary published January 18, 1988, Prof. Ching Y Suen, Concordia University, Montreal, Canada. Comments included why most handwriting character recognition accuracy reports are "inaccurate" [Summagraphics87] Summagraphics "Digitizer technology paves the way for portable systems", Computer Design, April 1, 1987, page 9 Summagraphics' low-cost charge-ratio digitizer tablet [TaguchiY87a] Taguchi, Yoshinori and Yamanami, Tsuguya "Position Detecting Device", United States patent 4,704,501, November 3, 1987 Electromagnetic grid digitizer, active stylus [Tamura88] Tamura, S. and Kawasaki, S. "Recognition of Sign Language Motion Images", Pattern Recognition, Vol 21 No 4, 1988, pp Cheremes - like phonemes or graphemes, but for (hand) sign language, like American Standard sign language [TannerPP87a] Tanner, Peter P. "Multi-Thread Input", Computer Graphics, Vol 21 No 2, April 1987, pp User interface discussion of having multiple input devices / multiple streams of input: relates to problem of handwriting combining both position AND command/text input, as in gesture recognition? [TappertCC87a] Tapper, C.C., Suen, C.Y., Wakahara, T. "Research Report: On-Line Handwriting Recognition - a Survey", Research Report RC (#59748) 12/8/87, IBM Research Division, T.J. Watson Research Center, Yorktown Heights, New York Report on state-of-the-art in handwriting recognition, pen-computing, gestures, curve-matching, elastic matching, etc. as of about 1986 Compare with 1990 publication: Tappert90c [TappertCC87b] Tappert, C.C., Fox, A.S., Kim, J., Levy, S.E., and Zimmerman, L.L. "Handwriting recognition on transparent tablet over Flat Display", Proc. of SID, Vol 28 No 1, 1987, pp Also Society for Information Display Digest of Technical Papers, Vol XVII, pp , May 1986 Cited in Rhyne86: Scriptel Digitizer Rhyne86 cites other version on visual parallax for electronic ink: 0.06" lateral shift causes subject's discomfort. Tappert describes also slippery surface. Cites internal Pencept report by Phillips and Prentice. [TappertCC88a] Tappert, Charles C. "Recognition System for Run-on Handwritten Characters", United States Patent 4,731,857, March 15, 1988, assigned to International Business Machines Corporation, Armonk, New York. Character stroke segmentation/parsing using an iterative approach to find the best aggregate recognition score for the whole line, or for whole word. Recognition method for run-on handwriting repeating receiving, examining and gradings steps until all strokes have been processed and recognized. [TappertCC88b] Tappert, C.C. "A Divide-and-Conquer Cursive Script Recognizer", IBM Res. Rep. RC14070, Oct Cited by Marlin Eller, Microsoft Pen Computing group [TappertCC88c] Tappert, Charles C. and Wakahara, Toru "On-line Handwriting Recognition - a Survey", Proc. 9th Int'l. Conf. on Pattern Recognition, November 14-17, 1988, pp vol 2 8:26:29 PM]

161 ( ) History of Pen and Gesture Computing Overview of handwriting recognition, including wild point correction, dot reduction, stroke connection (extraneous pen lifts), etc. Gesture as hand markings, pairwise comparison of functional attributes. Long list of commercial system circa 1988 (Pencept, Quest, CIC, DES, etc.). See also 1990 journal paper. [Tektronix87a] Tektronix "Tektronix Smalltalk Users Manual", Tektronix, Part No Smalltalk language and graphical user interface: describes draw-through or drag gesture for selecting text, in addition to single/double/triple click to select words, sentences, etc. Selection for click also depends on where you click: in middle of word or at start of line. [TerryJA88a] Terry, J.A. and Hsiao, H. "Tactile feedback in a computer mouse", Proc. 14th Annual Northeast Bioengineering Conf., March , 1988, pp (abstract only) Mouse with piezoelectric transducer producing vibrotactile feedback / haptic vibrations to tip of index finger when cursor arrives at target. [TeulingsHL88a] Teulings, Hans-Leo "Handwriting-Movement Control: Research into Different Levels of the Motor System", Ph.D. dissertation, Katholieke Universiteit Nijmegen, The Netherlands, 1988 study of handwriting motor control. Instrumentation used Vector General Data Tablet DT1, "RMS error less than 0.2 mm, 200Hz". Review of handwriting angle/tilt, grasp of pen. Long stroke dominated by fingerjoint movements, short strokes by wrist joint movements. Compare with Kuklinski/Ward variability model? [TexasInstruments87a] Texas Instruments "Texas Instruments Business Edge Guidebook", Texas Instruments, 1987 Hand-held calculator with PowerTouch display, five touchpad buttons whose labels and functions change to set different functions when user hits SCROLL key. File contains additional materials. Also sold as Radio Shack EC [TexasInstruments88a] Datamath Calculator Museum "Texas Instruments Business Edge Guidebook", Texas Instruments, datamath.org/sci/modern/businessedge.htm Hand-held calculator with touchscreen display, shows labels of functions under function key spots on display. Feature also found in TI-81 (variable function keys), Texas Instruments Personal Banker. [TransImage87] TransImage Corporation "Transimage 1000", hand-held OCR data-entry system for IBM PC XT/AT product announcement, 910 Benicia Avenue, Sunnyvale, California 94086, 1987 Hand-held optical scanner for OCR: M processor board for IBM PC XT/AT (compare Pencept and CIC) [TsaiRY87a] Tsai, Roger Y. "A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-shelf TV Cameras and Lenses", IEEE Journal of Robotics and Automation, Vol RA-3, No 4, August 1987, pp Mathematical method for calibrating a camera (estimating distance to object, location of object, pincushion or other well-behaved distortion of lens) in 3D space using views from a single camera. Uses known 2D target pattern with known points. Contrast with structure-through-motion. [TsudaI88a] Tsuda, I., Shimizu, H., and Hibino, K. "Method and Apparatus of Recognition", United States Patent 4,760,603, July 26, 1988, assigned to Research Development Corporation of Japan, Tokyo, Japan. Strange patent on recognition by looking at oscillation frequencies of hypothetical components stretched between segments of the outline image [Tsugei87] Tsugei, Shinji and Iguchi, Shigeki "Handwritten Character Input Device", United States Patent 4,656,317, April 7, 1987, assigned to Sharp Kabushiki Kaisha, Osaka, Japan. Handwriting recognition for processing billing receipts on-line: patent on any kind of forms input? [UngarD87] Ungar, David and Smith, Randall B. "Self: The Power of Simplicity", OOPSLA '87, pp Description of SELF programming language: prototype-based object-oriented language: prototypes, slots, and behavior. Prototypes combine inheritance and instantiation. Walter Smith said this was the base for NewtonScript. Cites to Smalltalk. No discussion of SDK. [Viglione88] Viglione, Sam S. "Voice input systems", Computer graphics: technology and applications, Academic Press, Boston, 1988 ISBN Cited in LCS/Telegraphics case vs. Schumer for input devices like virtual tablets [Wacom88a] Wacom "User Manual for SD-510 A5 Type Digitizer -- Provisional", Wacom, July 1988 Wacom small-format tablet digitizer, Summagraphics Bit Map Two emulation. Full list of Bit Pad Two commands (Summagraphics). Proximity is Z height at which coordinates are not correct. Batteryless / nobattery stylus and cursor (puck). [Wada87] Wada, Y., Kobayashi, Y., and Mitsuta, T. "Graphic data design method", European Patent , April 15, 1984, assigned to Hitachi, Limited, Tokyo, Japan. User interface patent on design rule checking (?) drawing constraints on CAD input (?) [WangA87] Wang, A., Ho S., and Mainemer, C.I. "Writing Pad", United States Patent 4,638,118, January 20, 1987, assigned to Wang Laboratories, Inc., Lowell, Massachusetts. Dr. An Wang's digitizer: force/pressure-sensitive digitizer using very thin grid of contact lines, as for LCD's. Force/pressure-sensitive digitizer using coarse, then fine position sensing of X/Y grid. Touch surface has 8:26:29 PM]

162 ( ) History of Pen and Gesture Computing special zones for special computer codes (commands). [WangLaboratories88a] Wang Laboratories "Video: Wang Freestyle Demonstration", Available on YouTube: video link to SIGGRAPH videos from Want Freestyle desktop stylus/handwriting system, documents may be annotated with dynamic electronic ink and real-time voice recording.. No handwriting recognition, extensive use of direct manipulation gestures. Dwell/press-and-hold. [WangPSP88] Wang, P.S.P. "Knowledge Pattern Representation of Chinese Characters", International Journal of Pattern Recognition and Artificial Intelligence, Vol 2 No 1, 1988, pp Not really about a Chinese/Kanji recognition system, but that Chinese characters are complicated (lots of strokes) because they are like English words (lots of strokes from lots of characters), yet are superior in many ways to Roman/Latin alphabets, and keyboard input technologies for Chinese ideograms have been worked out. [Ward87a] Ward, J. "Design Criteria for an Electronic Tablet Technology with Acceptable Performance for Handwriting Capture and Analysis", Proceedings of the Third International Symposium on Handwriting and Computer Applications, Montreal, Canada, July 20-23, 1987, pp [Ward87b] Ward, J. "Issues in the validity of testing protocols and criteria for on-line recognition of handwritten text", Proceedings of the Third International Symposium on Handwriting and Computer Applications, Montreal, Canada, July 20-23, 1987, pp [Ward87c] Ward, J. and Phillips, M. "Digitizer Technology: Performance Characteristics and the Effects on the User Interface", IEEE Computer Graphics and Applications, April 1987, pp Basic tutorial on tablet digitizer / touchscreen performance and errors: Gaussian noise, wild data, slew rate errors. Mutability of tablet performance and "specsmanship" on specifications. Overview of touchscreens and tablets: grid/sheet, electromagnetic/emr, electrostatic/capacitive, resistive, sonic. Missing coordinates. Tilt error. Mechanical slop. [Ward87d] Ward, J. (organizer): Sibert John (Moderator): Buffa, Michael G., Crane, Hewitt D., Doster, Wolfgang, and Rhyne, James "Issues Limiting the Acceptance of User Interfaces Using Gesture Input and Handwriting Character Recognition", panel discussion, Proceedings of CHI+GI Conference on Human Factors in Computing Systems and Graphics Interface, Toronto, April 5-9, 1987, pp Much work on handwriting character recognition is very over-stated Participants from Nestor, SRI International, CIC, AEG Research Institute, IBM Thomas J. Watson Research Center, Pencept [Ward87e] Ward, J. "Combinatorial Aspects of Variability in Handprinted Text", Proceedings of the Third International Symposium on Handwriting and Computer Applications, Montreal, Canada, July 20-23, 1987, pp [Ward88] Ward, J. and Kuklinski, T. "A Model for Variability Effects in Hand-printing, with Implications for the Design of On-line Character Recognition Systems", IEEE Trans. Systems, Man, and Cybernetics, Vol 18 No 3, May 1988, pp Generative model for shape variations in handprinted and handwritten characters, based on stroke order, angle, hooks, retracing, and a modification of chain codes using local extrema, for on-line handwriting character recognition [WareC87a] Ware, Colin and Mikaelian, Haratune H. "An Evaluation of an Eye Tracker as a Device for Computer Input", Proc. CHI + GI 98, pp Eye tracker better than mouse or tablet as pointing device, because do not need to find location on screen, then move eyes to mouse, then back. (Not sure about that!) Mentions dwell time for selection / mouse click: compare with press-and-hold. Gulf and Western series 1900 eye view monitoring system, determines boundary of pupil. [WareC88a] Ware, Colin and Jessome, D. "Using the Bat: A Six-Dimensional Mouse for Object Placement", Graphics Interface '88, 1988, pp Bat: Using Polhemus Isotrack 6 DOF three-dimensional electromagnetic digitizer as input device for 3D placement: uses relative motion / relative displacement (like a mouse) instead of absolute position, so distortions from nearby metallic objects etc. no a factor. [WatanabeY88a] Watanabe, Yasuhiro "Position Control Apparatus", United States Patet 4,734,685, March 28, 1988 Scrolling a viewport on a larger image with a mouse, touchscreen, or other pointing device. Viewport continues sliding. Has minimum distance check: press-and-hold? [WeingartSH87a] Weingart, Steve H. "Physical Security for the µabyss System", Proc IEEE Symp. on Security and Privacy, April , 1987, Oakland, CA, pp Physical security of processors: sealed processor and memory, basically booby-trapped (embedded wires, short out if package drilled, break if opened etc.) if tampered with physically. Notes that it does not address freezing of CMOS memory. [WelbournLK88a] Welbourn, L.K. and Whitrow, R.J. "A gesture based text editor", Proceedings of the British 8:26:29 PM]

163 ( ) History of Pen and Gesture Computing Computer Society, pp , Cambridge University Press, 1988 Cited in Goodisman91, FordDM91a [WhiteSR87a] White, Steve R. and Comerford, Liam "ABYSS: A Trusted Architecture for Software Protection", Proc Symp. Security and Privacy, Oakland, CA, Apr , 1987, p Software copy protection / rights authorization based on passing rights-to-execute vi a token. Enforcement by protected/secure (logically, physically, procedurally secure) processors, encrypted/secure communications between secure processors. Rights-to-execute can be time-limited, dependent on callers, etc. Use count enforced by limited number of use-once rights-to-execute. [Whitrow87] Whitrow, R. and Higgins, C. "The Application of n-grams for Script Recognition", Proceedings of the 3rd Symposium on Handwriting and Computer Applications. Montreal, Canada, 1987 Cited by Marlin Eller, Microsoft Pen Computing group [Williford88] Williford, John D. "Automatic Reference Adaptation During Dynamic Signature Verification", United States Patent 4,724,542, February 9, 1988, assigned to International Business Machines Corporation, Armonk, New York. Continuous adaptive signature verification recognition: samples are updated automatically over time [Wolf87a] Wolf, Catherine G. and Morrel-Samuels, Palmer "The Use of Hand-drawn Gestures for Text-editing", International Journal of Man-Machine Studies, Vol 27 No 1, July 1987, pp , also available as IBM Report RC Paper and pencil study of gesture user-interface for editing (proofreader's marks) for user consistency, choice of symbols [Wolf87b] Wolf, Cathy G. "A comparative Evaluation of Gesture and Conventional Interfaces", IBM Research Division Human Factors research report RC13187 (#58999), October 9, 1987 Cited in Blatt88 gesture user interface, single stroke gesture for spreadsheets, with text and pointing recognition [Wolf88] Wolf, Catherine G. "A Comparative Study of Gestural and Keyboard Interfaces", Proceedings of the Human Factors Society 32nd Annual Meeting, 1988, pp Gesture user interface on spreadsheets: 72% faster than keyboard alone, preferred by users, using electronic ink integrated tablet/display: shows "preferred" gesture symbols [Xerox87a] Xerox Corporation, Vista Laboratory "Analyst V1.2t Reference Manual", Xerox Special Information Systems, Vista Laboratory, 1987 popup/pop-up menus on three-button mouse for various functions on single-click/double-click. Pop-up menu is context sensitive, depending on what is clicked on and whether there is a current selection. Implemented in Smalltalk-80. Draw-through for drag selection of text. [Yamakawa87] Yamakawa, Tadashi "Character and Figure Processing Apparatus", United States Patent 4,672,677, June 9, 1987, assigned to Canon Kabushiki Kaisha, Tokyo, Japan. User-interface combination of handwriting recognition, and simulated touch-button areas on tablet for Chinese/Kanji word processing Shows a menu of multiple candidate character to select from when the online recognizer for handwritten character isn't sure. [Yoshida88] Yoshida, K., Shimizu, H., and Watari, Massao "Continuous Characters Recognition System", United States Patent 4,764,972, August 16, 1988, assigned to NEC Corporation, Tokyo, Japan. On-line character recognition using inter-stroke handwriting strokes Refers to changes in separate handwritten character variations depending on inter-stroke motion (hooks, continuation marks, etc.) Refers to partially-connected (script-like) writing as continuous writing (don't stop between characters) [ZimmermanTG87a] Zimmerman, T.G., et al "A hand gesture interface device", Proceedings of the ACM CHI + GI '87 Conference on Human Factors in Computing Systems and Graphics Interface, Toronto, Canada, April 5-9, Association for Computing Machinery, New York, 1987, pp Cited in Nielsen90 Z-glove, Date-Glove, Data-Glove using ultrasonic/acoustic position sensors for threedimensional digitizing. Alternative input was magnetic sensing of small magnets on a glove. [ZingherO87a] Zingher, Oded "Data Input Unit and Method for Printing Machines", United States Patent 4,639,881, January 27, 1987 Transparent matrix switch / touch screen, using transparent ink (gold, etc.) on two sheets of mylar in row/column geometry. biblio86 (Prev page) Main Page (Next page) biblio90 All the good lucky numbers are already taken. Die Wahrheit ist wie ein Spiegel: den Spiegel selbst sieht man nicht, es zeigt nur die Welt, wie sie ist. 8:26:29 PM]

164 ( ) History of Pen and Gesture Computing 8:26:29 PM]

165 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1989 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:35 EDT Mirrors of this page can be found at: (Generally most up-to-date) biblio88 (Prev page) Main Page (Next page) biblio93 [ATT90a] AT&T "UNIX System V Release 4 Programmer's Guide: XWIN Graphical Windowing System Xlib-C Language Interface", Prentice Hall, 1990 Administrative / programming guide for XWIN/X-Windows windowing system on Unix. ACL access control list for network connection, which system programs may be run from. Access rights/permissions can be conditioned also on authorization name and data received at server end of connection. [AbadiM90a] Abadi, Martin; Burrows, Mike; Kaufman, Charlie; and Lampson, Butler "Authentication and Delegation with Smart-Cards", Technical Report SRC-RR0-67, 1990; Proc. TACS '91 International Conference on Theoretical Aspects of Computer Software, September , 1991, Sendai, Japan, pp Security and performance analysis of smart-card authentication protocols relying on public/private key encryption/decryption in the smartcard: neither smartcard PIN nor private key ever exposed to login station/computer. [Abramovitz89] Abramovitz, R.G., Anderson Jr., R.L., Busse, F.L., Celarier, S.A., Poublan, J-Y. H., and Shorthill, L.R. "Optical Character Reader", United States Patent 4,809,351, February 28, 1989, assigned to SABA Technologies, Incorporated, Beaverton, Oregon. Hardware design of a hand-held scanner for optical (typed) character recognition [AccessDevelopment90] Access Development Corporation "FormLink product literature", July, Address: 2000 Commonwealth Avenue, Auburndale, Massachusetts, 02166, Forms package, stylus-keyboard electronic-paper user interface, suitable for adding handwriting recognition [ActiveBook89] Active Book Ltd., Cambridge England "Masterstroke: the notebook computer", London Times, September 7, 1989, p 34 Portable touch-screen/handwriting product, with character recognition. Active Book: Dr. Herman Hauser, founder: Olivetti financing [AdolfJ89a] Adolf, Jurine "An Expert System for the Selection of Input Devices", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Select input/pointing devices -- Says on-line recognition inferior to almost all other user-interfaces for text input (comparison with Linus) 8:26:33 PM]

166 ( ) History of Pen and Gesture Computing [AdvancedProducts89] Advanced Products and Technologies "Voice Computer product announcement", IEEE Computer, September 1989, p 98 Portable voice (!) recognition and output computer product [AffinitoFJ89a] Affinito, F.J. "Braille Computer Mouse with Tactile Position Feedback", IBM Technical Disclosure Bulletin, Vol 31 no 12, p. 386, May 1989 Not a Braille-output display: mouse has pistons (raisable braille dots) to give haptic feedback (raised dots) as user moves mouse over display. No discussion of relative/absolute positioning/pointing. [Agilis89a] Agilis Corporation "Product information on the Agilis System (TM) of Handheld Workstations", June 12, San Antonio Road, Mountain View, California Portable touch-screen/handwriting product, but no character recognition or electronic ink user-interface: radio net, Unix/OS-2/DOS File includes additional material: BYTE magazine, "Agilis Hand-Held Workstations: Computing Power in the Field" Aug [Agilis89b] Agilis Corporation "Agilis Hand-Held workstations: Computing Power in the Field", BYTE Magazine, August, 1989 Portable touch-screen/handwriting product, but no character recognition or electronic ink user-interface: radio net, Unix/OS-2/DOS File includes additional material: BYTE magazine, "Agilis Hand-Held Workstations: Computing Power in the Field" Aug Hand-held Microsoft PC with modular disk drive module/slice, touchscreen/tablet display, battery packs, etc. [AkihikoM90a] Akihiko, Matsuo; Masume, Hanaeda; nd Atsuko, Tokuhara "Window Display System for Multiwindow System", Japanese Patent JPH (a), April 5, 1990 (Machine translation to English) Translucent/semi-transparent windows so that user can see objects underneath. [Apple89a] Apple Computer, Inc. "HyperCard User's Guide", Apple Computer, Copyright 1989, Cupertino, CA (hardcopy book) Basic reference for Hypertalk: new buttons may be defined to auto-highlight, includes lasso gesture for graphical editing [Apple89b] Apple Computer, Inc. "HyperTalk Beginner's Guide: An Introduction to Scripting", Apple Computer, Copyright 1989, Cupertino, CA (Hardcopy book) Basic reference for Hypertalk: describes tool palettes for buttons, text, selection: lasso selection gesture triggered by tool palette button [Apple90a] Apple Computer, Inc. "MacIntosh Reference", Apple Computer, Copyright 1990, Cupertino, CA (hardcopy book) Reference manual/user's guide for MacIntosh computer: CloseView screen magnifier for accessibility [Apple90b] Apple Computer, Inc. "Getting Started with your MacIntosh", Apple Computer, Copyright 1990, Cupertino, CA (hardcopy book) Reference manual/user's guide for MacIntosh computer: desk user interface, explains that "press" does not mean press on the mouse, it means press the mouse button [Apple90c] Apple Computer, Inc. "Getting Started with HyperCard IIGS", Apple Computer, Copyright 1990, Cupertino, CA Reference/tutorial for HyperCard: hyperlinked document cards. Examples of IDE/SDK. Mentions HyperTalk scripting language. [Apple90d] Apple Computer, Inc. "HyperTalk Beginner's Guide for the Apple IIgs", Apple Computer, Copyright 1990, Cupertino, CA Reference/tutorial for HyperTalk: scripting language for HyperCard with hyperlinks. Scripting language similarities to JavaScript (on mouseup, etc.) [AppleProgrammersAndDevelopersAssociation89a] Apple Programmer's and Developer's Association "Hypercard Developer's Toolkit Version 1.0", Apple Programmer's and Developer's Association, 1989 Apple Hypercard documentation from users group. User interface visual effects (scroll or wipe/swipe). AppMaker Application Generator with GUI graphical user interface editor. [AppleProgrammersAndDevelopersAssociation89b] Apple Programmer's and Developer's Association "ResEdit Version 1.2B1", Apple Programmer's and Developer's Association, 1989 Apple Hypercard document from users group. Resource Editor IDE, shows resource built in a file, resource picker. Properties and attributes of individual resources can be edited. MacsBug 6.0 debugger. Includes LightSpeed C: GUI of build utility allows files to be bundled into segments graphically (partition line) for both source and library files (resources also?) [Atari90] Atari Corporation "Portfolio hand-held personal computer", Sunnyvale, California, , 1990 Hand-held (well, hand-holdable) clamshell computer, no touchscreen. MSDos Includes calendar, diary, note taker, other PDA functionality. [AtkinsonR89a] Atkinson, Russ; Demers, Alan; Hauser, Carl; Jacobi, Christian; Kessler, Peter; and Weiser, Mark "Experiences Creating Portable Cedar", Xerox PARC Palo Alto Research Center, Report CSL-89-8, June Also 8:26:33 PM]

167 ( ) History of Pen and Gesture Computing Proc. SIGPLAN '89, Portland Oregon, June 21-23, 1989 Portable compilation of Cedar (successor to Smalltalk) by compiling to "C" and using native "C" compiler, plus implementing part of Cedar in the ported subset. X-Windows client. [Autosig90] Autosig Systems Inc. "Sign/on product information", 1990, PO Box 16050, Irving Texas 75016, Formerly Signify / Sign-on: signature verification product [BarkerBA89] Barker, Barbara A., Hernandez, Irene H., and Machart, Beverly H. "In-line dynamic editing of text or graphics", United States Patent 4,815,029, March 21, 1989, assigned to IBM Corporation. User interface combining free-hand drawing, graphical editing, similar to "Polite" product system by Sheldon Borkin and John Prager, or Xerox Star and Apple MacIntosh with generic functions valid for all types of objects On-line dynamic editing of text or graphics in computer document executing actions on user-selected attributes based on internal knowledge of object type rather than by explicit commands [BarkerBA90] Barker, Barbara A.; Hernandez, Irene H.; Machart, Beverly H. "Method for Modifying Intermingled Text Object and Graphic object within an Object Set Individually or Correspondingly", United States Patent 4,974,194, November 27, 1980 User-interface for stylus/gesture editing of text, using mark-up gestures [BarkerPG89] Barker, P.G. and Manji, K.A. "Pictorial dialogue methods", International Journal of Man-Machine Studies, Vol 31 No 3, pp , September 1989 User-interface for electronic books: multi-media pointing a la hypertext with pictures [BatsonB89a] Batson, Brian and Diehm, Brian D. "Touchscreen Feedback System", United States Patent 4,821,030, April 11, 1989 touchscreen user interface UI for oscilloscope, using touch zones to select regions of trace. Characterizes touchscreens as being LED optical. Considerable treatment of "debounce" due to slow scanning rate of optical sensors, may encounter finger during vertical scan but not horizontal scan. Long dwell time. [BaumE89a] Baum, E. and Haussler, D. "What size net gives valid generalization?", in Advances in Neural Information Processing Systems, D.S. Touretzky, editor, Morgan Kaufman, 1989 Cited in MartinGL90b: Trainable feedforward/back-propagation neural networks, mathematical derivation of size network needed to classify from a size of training sample set with particular expected error bounds. Notes however that this presumes the training set is not biased / is representative. [BeardMH90a] Beard, Marian H.; Caro, Perry A.; Hsiao, Jennifer B.; Mackey, Kevin J.; Sandman, James G.; Steinbach, Gary R.; and Wood, Donald R. "Virtual and emulated objects for use in the user interface of a display screen of a display processor", United States Patent 4,939,507, July 3, 1990 Simulated devices (PCs with simulated/virtual floppies, BIOS etc. etc.) also simulated/emulated virtual printers, etc. in a GUI desktop that is not a PC/DOS (Xerox). Includes a pop-up virtual keyboard that can be moved on the display. Mouse or (other) cursor control device (e.g. touchscreen). [BegejS90a] Begej, Steve "Glove Controller With Force and Tactile Feedback For Dexterous Robotic Hands", Phase I SBIR Final Report, 14 September, 1990, NAS Prototype for data-glove-like hand controller with tactile/haptic feedback to fingertips and joints. Intended as telepresence (compare virtual reality) controller, remote robotic control. Sensors for joint angles and joint force, 37-taxel tactile touch sensor array in fingertip, tactile display is inflatable (air) fingertip. [BehrensH90a] Behrens, Herbert and Zeising, Norbert "Arrangement for the Input and Processing of Characters and/or Graphic Patterns", United States Patent 4,899,137, February 6, 1990 Overlay mechanical/physical keyboard on tablet: keys press down on tablet. When keyboard cover lifted, can use tablet as touchpad. No description of actual touchpad sensor: cites to systems that would no do multi-touch -- no discussion of key rollover. Refers to pressure, but only as pressure causing touchpad switches to close, etc. [BeringerDB89a] Beringer, Dennis B. and Bowman, Mary James "Operator Behavioral Biases using High-resolution Touch Input Devices", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Nothing study of visual parallax and targeting error in integrated tablet/display: 3/4mm error on their hardware at 17 degrees viewing angle: software offset only good for one user in one position, so they suggest letting the user just get used to visual parallax [BierEA90a] Bier, Eric A. "Snap-Dragging in in Three Dimensions", ACM SIGGRAPH Computer Graphics, Vol 24 No 2, Mar 1990, pp Proc. I3D '90 Symp. on Interactive 3D Graphics Snap to grid vs. heuristic constraints in three dimensions 3-D similar to compass and ruler for pointing and positioning, in technical illustration system. Gravity, alignment objects (lines, angles, spheres, etc.), smooth-motion affines. Snap-to-grid in three dimensions in 2-D user interface. [BierEA90b] Bier, Eric A. and Goodisman, Aaron "Documents as User Interfaces", Proc. Int'l. Conf. on Electronic Publishing, Document Manipulation and Typography, 199, pp Earliest paper on EmbeddedButtons. Mentions navigation buttons embedded in electronic document 8:26:33 PM]

168 ( ) History of Pen and Gesture Computing (buttons in TOC and in section headings), turn words into buttons that do strike-out or other function (translation/dictionary/spelling-change) when clicked. Buttons also embedded in multimedia documents. Video demonstration. [BierEA90c] Bier, Eric A., Goodisman, Aaron; Wyatt, Doug; ndpier, Ken "Documents as User Interfaces: Video", Xerox PARC demonstration video for EmbeddedButtons, 1990, Available on YouTube.com Demonstration video for EmbeddedButtons: portions of text in a document automatically made active and clickable: e.g. bring up dictionary definition, show a pop-up menu, etc. Tioga text editor, Gargoyle graphics editor, Documents as Control Panels. To accompany 1991 SIGCHI paper. Compare with Visual Basic, PenApps? [BilboMK90a] Bilbo, Mark K. "Using MacWrite: Covers MacWrite II", Que Corporation, 1990 (hardcopy book) MacWrite II word processing / electronic publishing. Find/Replace uses wildcards, then presents user multiple change actions [BlattLA89a] Blatt, Louis A. "Human Factors Support of Gesture Recognition", research report June 16, 1989, Wang Laboratories, 1 Industrial Avenue, Lowell, Massachusetts Internal Wang report on variations in chain code writing sequences for handwritten alphabetic characters [BlattLA89b] Blatt, Louis A. "Spreadsheet Gesture Set", research report February 2, 1989, Wang Laboratories, 1 Industrial Avenue, Lowell, Massachusetts Internal Wang report on what editing marks / gestures subjects used informally on spreadsheets [BlattLA90a] Blatt, Louis A., and Schell, Alan "Gesture Set Economics for Text and Spreadsheet Editors", manuscript, March 3, 1990, Wang Laboratories, 1 Industrial Avenue, Lowell, Massachusetts Gesture recognition UI study on "most natural" gestures for text and spreadsheet editing, resulted in proofreader's marks [BlehaS90a] Bleha, Saleh; Slivinsky, Charles; and Hussien, Bassam "Computer-Access Security System Using Keystroke Dynamics", IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol 12 no 12, December 1990, pp Biometric Authenticatiion of users by keystroke/typing dynamics. Identifier uses a phrase, password uses person's name. 8% false reject, 0.5% false acceptance. Citations to signature handwriting authentication/verification, but none to other keyboard dynamics. See Saltzer 1976 [BloodEB89a] Blood, Ernest B. "Device for quantitatively measuring the relative position and orientation of two bodies in the presence of metals utilizing direct current magnetic fields", United States Patent 4,849,692, July 18, 1989 Ascension Technology Corporation: electromagnetic 3D/6DOF position sensing (tablet) using multiple receiving antennas (orthogonally at right angles) and DC direct current magnetic fiel from multiple transmitters. Subtract out Earth's magnetic field. Vague about the exact computation of position: based on direction inferred from relative signal strengths along three axis, and distance inferred from absolute signal strength w.r.t. earth magnetic field as reference? Says with DC signals, no need for calibration (?), no interference from electromagnetic fields (?), but affected by nearby magnetic object. [BloodEB90a] Blood, Ernest B. "Device for quantitatively measuring the relative position and orientation of two bodies in the presence of metals utilizing direct current magnetic fields", United States Patent 4,945,305, July 31, 1990 Ascension Technology Corporation: electromagnetic 3D/6DOF position sensing (tablet) using multiple receiving antennas (orthogonally at right angles) and DC direct current magnetic fiel from multiple transmitters. Subtract out Earth's magnetic field. Vague about the exact computation of position: based on direction inferred from relative signal strengths along three axis, and distance inferred from absolute signal strength w.r.t. earth magnetic field as reference? Says with DC signals, no need for calibration (?), no interference from electromagnetic fields (?), but affected by nearby magnetic object. [BlySA90a] Bly, Sara A. and Minneman, Scott L. "Commune: A Shared Drawing Surface", Proc. COCS '90 ACM SIGOIS and IEEE CS TC-OA conference on Office information systems, Pages , ACM New York Shared writing/marking/drawing/gesture surface (whiteboard) with voice, transparent digitizer over display with single pen/stylus for each user. All user ink was same color, cursor "pens" different color for each user. Pointing by moving cursor without touching surface. Simultaneous by all users. Transparent digitizer not specified. Gestures were in-air for other users to see what speaker was pointing at, not gesture commands. (proximity sense digitizer: Scriptel?) Identify/differentiate users? Compare with Wang Freestyle? [BoieB89a] Boie, Bob; Mathews, Max; Schloss, Andy "The Radio Drum as a Synthesizer Controller", Proc. International Computer Music Conference (1989), pp Capacitive touch sensor: transmitting antennae in multiple drum sticks (e.g. multi-touch), receiver in flat drum sensors, different drum sticks use different frequencies. Z proximity sensing up to about 6 inches (compare with Scriptel). Bimanual/multi-touch [BoieRA89b] Boie, R. A.; Ruedisueli, L.W.; and Wagner, E.R. "Gesture Sensing via Capacitive Moments", Work 8:26:33 PM]

169 ( ) History of Pen and Gesture Computing Project No (2099,2399), AT&T Bells Laboratories, 1989 See also Blonder 93 patent involving capacitive moments, and Radio Drum. Cited in Jaffe 93 essay on electronic instruments, further electronic music references. Some references give the date as [BorensteinNS90a] Borenstein, Nathaniel S. "Multimedia Applications Development with the Andrew Toolkit", Prentice-Hall, 1990, ISBN (hardcopy book) Mostly Dialog and other aspects of Andrew UI programming [BorgK90a] Borg, Kjell "ishell: A Visual UNIX Shell", Pro. Chi '90, April 1990, pp Visual interactive shell user interface for UNIX using gestures: Desktop UI using gestures: drag-and-drop document icons on on UI machines such as printer, mailer, fax, etc. Visual programming language iscript for defining actions. [BowersDevelopment89a] Bowers Development "AppMaker, the Application Generator", Bowers Development, Lincoln MA 1989 ThinkC Class Library object-oriented development system for the Apple Macintosh -- forerunner to FrameMaker? Shows UI for creating dialogs on targeted system. GUI IDE for cross-system development. Includes addendum for version 1.1 of Appmaker. GUI IDE allows for custom sliders. [BowersHL90a] Bowers, Harold L. "Interactive Template", United States Patent 4,933,514, July 12, 1990 (Printed patent miss-spells Interactive Template): Patent on CadKey fixed-menu command template to be put on top of a digitizer with command iput, commands being grouped on the template. Compare with other templates of the time, such as PenCad, Pencept template with recognition macros (gestures), and work by Doster and Oed. [BozinovicRM89a] Bozinovic, R.M. and Srihari, S.N. "Off-Line Cursive Script Word Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence", Vol 11 No 1, January 1989, pp (abstract only) Off-line (vs on line) cursive handwriting recognition. Multi-level analysis of handwriting: points, contours, features, letters, and words Reference-line finding on handwriting, letter segmentation based on local minima or low vertical profiles Topological relationships of contours on handwriting for recognition Lists of features used for cursive handwriting recognition Chain encoding for off-line handwriting cursive script recognition Statistical and syntactic context rules for cursive handwritten words Testing done with two subjects told to do fully-connected script, and to maintain reasonable legibility Said 77% accuracy on whole words with context was good test results (!) Testing of off-line cursive script, but used data taken from tablet digitizer, so no scanning/pixel/gray problems Well-known tenet of pattern recognition practice that for problems with large feature space dimensionality, the sizes of the training and testing sets should be roughly the same Well-known tenet of pattern recognition practice that training and testing sets should not be the same to avoid overly optimistic results [BrooksFP90a] Brooks, Frederick P. Jr.; Ouh-Young, Ming; Batter, James, J. and Kilpatrick, P. Jerome "Project GROPE - Haptic Displays for Scientific Visualization", Computer Graphics, Vol 24 No 4, August 1990, pp Haptic output display visualization of molecules (electric field surface) and objects, with force-feedback device. GROPE-1 Device used developed by Margaret Minsky. [BrownE90a] Brown, E., Buxton, W., and Murtagh, K. "Windows on Tablets as a Means of Achieving Virtual Input Devices", in D. Diaper et al (Editors), "Human-Computer Interaction - INTERACT '90", Elsevier Science Publishers, pp See also Refers to dynamic windows, virtual devices on tablets, tools for defining regions flexibly. Use touch tablet to simulate keypads, mouse, other devices. Compare with Schumer? Mentions emulating multiple devices on multi-touch tablet (Lee '85) [BrownNR89a] Brown, Norman R. and Vosburgh, Ann Marie "Evaluating the Accuracy of a Large-Vocabulary Speech Recognition System", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Recognition performance: simple recognition error (how many wrong): adjusted recognition error (how many wrong after removing errors system "not expected" to get right) out of vocabulary words (also removing errors attributed to bad input: tablet or voice input errors? Use of context can ADD errors and reduce recognition accuracy: "drag-along" errors where a "correction" due to context screws up subsequent inputs User interface and recognition %: a (speech) recognition system need very good / easy correction mechanisms for user to correct recognition errors [BuxtonWAS90a] Buxton, William A.S. "A Three-State Model of Graphical Input", Proc. INTERACT 90, pp Modeling pointing (stylus, joystick, mouse, touchscreen, lightpen) user interfaces as transitions among three states: touch, proximity, out-of-range. Does not model multiple buttons directly. Tap is no motion in touch. Mentions timeout cue (no motion/dwell gesture) for pointing. Dwell combine with press-and-hold? [ByrneJB89a] Byrne, Jani Gabriel "Competitive Evaluation in Industry: Some Comments", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Notes on comparing competitive commercial user interfaces [Byte89a] BYTE Magazine "BYTE Magazine: Annual Awards and digitizing Tablet", BYTE Magazine, Vol 14 no 1, 8:26:33 PM]

170 ( ) History of Pen and Gesture Computing January 1989 "Product Focus: Graphic Details" by Apiki, Steve and Diehl, Stanford. Includes review of Penpad 320 of Pencept: show character recognition CAD templates (AutoCad), controller board. [CIC89a] Communication Intelligence Corporation "Handwriter: Expanding the world of computing", CIC, 1989 Product and technology brochure on CIC Handwriter: ideographic writing, non-typed. Several examples of program templates for desktop publishing, none for CAD/CAM. Comparison of user interface on touch tablet with single touches, versus multiple keystrokes for spreadsheet programs Lotus/123 and Multiplan. [CIC89b] Dao, Jeff "Handwriter: The Ultimate Recognition Engine", Communication Intelligence Corporation, 1989 Overview on CIC Handwriter: Chinese/Japanese characters handwriting recognition, CAD/CAM, Office Automation (Templates for WordPerfect, Lotus 1-2-3, etc.) Automatic data processing (forms input), Small Business Accounting (Solomon III) [CIC90a] Communication Intelligence Corporation "Communication Intelligence Corporation: Handwriter Licensing program for Wang Laboratories", CIC, April Licensing program for CIC HandWriter handwriting/gesture recognition on tablet: compare with Pencept. [CIS89] CIS "Geometry Ball product information", CIS Group, Postfach , D-4060 Viersen 1, West Germany, DOF Six-degree of freedom 3-D pointing mouse device: senses torque (yaw, pitch, roll) and direction of force (X/Y/Z), motionless joystick. [CalComp89a] CalComp "CalComp 2300 Series DrawingBoard Digitizers", CalComp Incorporated, 1990 CalComp 2300 digitizer tablet, stylus, cursor/puck. File includes brochure on CalComp graphic input devices [Canon89] Canon "Ai Note IN-3000 product announcement", CanonVoice, February, 1989 (in Japanese). Small, portable notebook-style handwriting input/recognition (PDA) product from Canon, 1980 [CardSK90a] Card, Stuart K.; Mackinlay, Jack D.; and Robertson, George G. "The Design Space of Input Devices", Proc. CHI '90, pp Comparison of user-interface features for tablets, touch pads, light pens, mouse, and other pointing input devices: resolution, foot print, speed and error performance, accuracy. [CarpenterG89a] Carpenter, Gail and Grossberg, Stephen "Hierarchical neural approach", Boston University press release, 1989 Report of ongoing research for multi-layer neural networks with back-propagation [ChengF89] Cheng, F., Hsu, W., and Chen, C. "Fuzzy Approach to Solve the Recognition Problem of Handwritten Chinese Characters", Pattern Recognition, Vol 22 No 2, pp , 1989 Fuzzy set recognition applied to Chinese characters Most troublesome problem in Chinese handwriting is great variation among hand-prints Assumptions about statistical probability density in handwriting recognition are highly dubious It is impractical to deal with variations in handwriting with probability densities A similarity measure between strokes in Chinese character based on angle feature, not length Stroke length not a reliable feature in Chinese characters 96% recognition rate on 881 Chinese characters (160 variations each), once rejects are removed [ChowD90a] Chow, Doris and Kim, Joonki "Paper Like Interface for Educational Applications", Proceedings of National Educational Computing Conference, June 20-22, 1989, Boston, Massachusetts, pp Part of the paper-like interface PLI projects at IBM: electrostatic/capacitive digitizer (Scriptel) over LCD display, exploration of gesture input. Teaching block-printing to children by direct comparison with prototype shapes. States that gesture recognition requires a different recognition strategy. Crosswordpuzzle as teaching drill using handwriting recognition. Music editing demonstration modelled after Buxton, but not using same recognition technology. [ClarkAJ90a] Clark, Andrew J. "Do you really know who is using your system? - A Survey of Personal Authentication Techniques", Paper to be presented to the British Computer Society, Technology of Software Protection Specialist Group, February 1990 Review of authentication techniques: Something Known (password), Something Owned, Optically read cards, Smart cards, Password Generators, Personal Authenticators, Physical Characteristics -- Fingerprint, Retinal Scan, Voice Verification. Mentions *dynamic* signature verification as an "involuntary response", cites low hardware reliability of touch-screens: Dr. John Parks of the British Technology Group for signature verification products [ClarkD89b] Clark, D.; Demmel, J.; Hong, J.; Lafferriere, G.; Salkind, L. and Tan, X. "Teleoperation Experiments with a Utah/MIT Hand and a VPL DataGlove", NASA/JPL Space Telerobotics Conference, JPLPublication 89-7, Vol. V, January 1989, pp DataGlove with articulation sensors combined with Polhemus 3D digitizer sensor used for Waldo-like teleoperation of robotic hand. [ClarkDM89] Clark, D.M. "Pattern Recognition Method and Apparatus", United States Patent 4,805,225, February 14, 1989, assigned to The Research Foundation of the State University of New York, Albany, New York. Hierarchical network recognizer, similar to neural nets, to determine the presence of a feature (but how 8:26:33 PM]

171 ( ) History of Pen and Gesture Computing are features weighted?). Cites work on connectionist models for neural nets [ComputerBusinessReview89] Computer Business Review "IBM Announcements - Computer Business Review", Computer Business Review, Issued 1146, March 31, 1989 Report on IBM PLI "paper-like interface", and comparison with Quest Automation Datapad development in the United Kingdom: Quest Datapad 15 years previously (1975?). [ComputerS89] Computer Shopper "Paper-like Interface -- IBM researchers are exploring a way to use computers in which people simply write on a flat surface", Computer Shopper, June 1989, page 169 Press announcement of IBM's electronic ink handwriting recognition product [CorrentDR89a] Corrent, Dean R. "Computer System User Interface", United States Patent 4,827,410 User interface for transparent digitizer or touchscreen which covers a display and also portions of working surface next to the display. Fixed Menus can be mounted outside the area of the display. [DFM89a] DFM Business Systems Inc. "DBF unveils 3.5-pound hand-held PC", Travelite 286 Product announcement, Info World, September 11, 1989, vol 11 no 37 [DFM89b] DFM Business Systems Inc. "Hand-held CPU for health care uses infrared-link", Travelite 286 Product announcement, Network World, Oct 30, 1989, v6 no 43 [DanisCM89a] Danis, Catalina M. "Developing Successful Speakers for an Automatic Speech Recognition System", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Training users to adapt to a particular speech recognition: Problems are too clear at end of words, too mushy at start of words, no word separation, speak too fast Data base /testing: very important to try realistic conditions: realistic conditions include the fact that users adapt to the system [DannenbergRB89a] Dannenberg, R.B. and Amon, D. "A gesture based user interface prototyping system", Proc. UIST '89, 1989, pp Slide-switch, two-finger (multi-touch) twist knob, click gestures. Hand-coded recognizer in prototype. Buttons: multitouch? Touchscreen/tablet was McAvinney Sensor Frame. Describes user interface system using object that are responsive to multiple, concurrent gestures (more than one finger); object is responsive to gesture if point of (first) contact falls within its bounding box. Includes "graphical switch" and "graphical slide switch" to turn on/of. [DarraghJJ90a] Darragh, John J.; Witten, Ian H. and James, Mark L. "The Reactive Keyboard: A Predictive Typing Aid", IEEE Computer, November 1990, pp Graphical/virtual/synthetic keyboard on-screen keyboard on Apple MacIntosh, usable with mouse without physical keyboard. Also includes predictive type-ahead. One use is as accessibility technology, combined with dwell-mouse for handicapped use. [Datakech90a] DataKech, Incorporated "Scriptwriter product information", Manor Village Lane, Rockville, Maryland [Datapro89] Datapro Research, Manufacturing Information Systems Data Collection report #450 "Handwriting Recognition Systems", January 1989 Datapro report on Linus Write-Top and Data Entry Systems ScriptWriter handwriting recognition products. [DavisE89a] Davis, Elliot "Pattern Recognition Error Reduction System", United States Patent 4,802,231, January 31, 1989 Error reduction by matching patterns against additional templates of patterns otherwise mis-matched. [Denker89] Denker, J.S., Gardner, W.R., Graf, H.P, Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., Baird, H.S., and Guyon I. "Neural network recognizer for Hand-written zip code digits", In Advances in neural information processing systems I, D.S. Touretzky, editor. Morgan Kaufman, 1989 Martin,GL90 [DestombesF89a] Destombes, F. "Virtual All-Points-Addressable Tactile Display", IBM Technical Disclosure Bulletin, May 1, 1989, Vol 89 No 5, pp Compare with Opticon: small finger-sized display for tactile output. Refers to small display being movable over desktop surface, and sensing absolute position on the surface (such as through mesh/grid of wires: electromagnetic/electrostatic tablet?) [DiSantoFJ89a] DiStanto, Frank J. and Krusus, Denis A. "Data/Facsimile Telephone Subset Apparatus Incorporating Electrophoretic Displays", United States Patent 4,870,677, September 26, 1989 Desktop telephone set with touchscreen display: displays contact lists/directory, different keypads, credit card and other information. Compare with Ken Knowlton? [DominessyME89a] Dominessy, Mary E. "A Literature Review and Assessment of Touch Interactive Devices", Technical Note 11-89, AD-A , U.S. Army Human Engineering Laboratory, Aberdeen Proving Ground, Md, October 1989 Review of touchscreen technologies and products Johnson Switch one-layer (horizontal only) wiregrid capacitive as first touch input device 1983/1967. Cross-wire capacitive/electrostatic. Resistive membrane. Capacitive film/sheet. Optical LED grid, optical LED from two corners. Elographics SAW surface acoustic wave with Z-axis force/pressure. Notes on parallax (esp. over curved CRT display), dead zones / 8:26:33 PM]

172 ( ) History of Pen and Gesture Computing target size (compare invention disclosure), [Dragon90] Dragon Systems "DragonDictate product announcement", Newton, Massachusetts, March 1990 Commercial voice/speech recognition product [DrummDE89a] Drumm, Donald E. "Electronic Light Pointer for a Projection Monitor", United States Patent 4,808,980, February 28, 1989, assigned to Wang Laboratories, Lowell, Massachusetts. Patent on wall-mounted light pen for projection TVs, displays [DulaneyKL89a] Dulaney, Kenneth L.; Hawkins, Jeffrey C.; Sanford, Lindsay; and McNamara, James H. "Split Screen Keyboard Emulator", United States Patent 5,031,119, July 9, 1991 On-screen simulated touchscreen keyboard, show full CGA display in top half of screen, on-screen virtual keyboard in bottom 1/2 of screen. [DunthornDI90a] Dunthorn, David I. "Virtual Button for Touch Screens", United States Patent 4,914,624, April 3, 1990 Get around lack of proximity sensing in virtual push buttons (with highlighting) on a digitizer touch screen, touch and untouch (sic) events. First touch/untouch does a selection, second touch/untouch actual activates what was selected. Instead add a second touch to the first touch, or move the finger in a particular direction, or press harder, and get a second button or a rotating menu to select/perform the particular action. [Dynabook89] Dynabook Technologies "product literature and press release", October PC/Computing, October 1989 Rumored portable touchscreen product ("Dynabook"): handwriting recognition? Any relation to Alan Kay's Dynabook? [EvansB89a] Evan, Blair "Method of and apparatus for sensing the location such as coordinates of designated points on an electrically sensitive touch-screen surface", United States Patent 4,806,709, February 21, 1989 Modification of capacitive sheet touchpad to use separated point electrodes to shape field, less aberration. Measurements are made among electrodes, rather than to ground. Can determine position outside the area of the electrodes. Refers to variations in moisture content of finger causing errors in capacitive reading. [EventoffFN89a] Eventoff, Franklin N. "Digitizer Pad", United States Patent 4,810,992, March 7, 1989 Resistive grid digitizer with force/pressure sensing, using a pressure sensitive layer between the two grids. One grid conductive, one grid resistive with resistance gradient. Grid need not be orthogonal, shows circular and trapezoidal grids. [Evergreen90] Evergreen Technologies "Gestures product announcement", Jeffrey Siegel, Fox Chapel Center, Suite 401, Frederick Road, Germantown, Maryland 20874, personal correspondence, March 12, 1990 Handwriting (adaptive?) character recognition product, gesture interfaces? [Exos90] EXOS, Inc. "Dexterous Hand Master exoskeleton product information", August Blanchard Road, Burlington, MA 01803, Data-glove, mechanical version, n-dimensional pointing device including Polhemus 3-D digitizer [Faggin89] Faggin, Federico, and Lynch, Gary S. "Brain Learning and Recognition Emulation Circuitry and Method of Recognizing Events", United States Patent 4,802,103, January 31, 1989, assigned to Synaptics, Incorporated, San Jose, California. Neural net patent Successive stages of recognition thresholds set for neural nets learning Includes neural net feature to resolve mis-recognized patterns confused with a pattern already learned Press announcement and description of IBM prototype signature verification system [FederalCW89] FCW "Smart signatures at FED: Net Prototype uses Biometrics, Smart Cards", Federal Computer Week, May 22, 1989, page 1 [FellowsGL89a] Fellows, Gerald L. and Freivalds, Andris "The use of force sensing resistors in ergonomic tool design", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Pressure/force sensors for hand-tools (drills, hammers, etc.): also stylus? Compare with FSR touchscreens. [FlemingSS90a] Fleming, S.S. "IBM Fleming Architecture for Separate User-Interface Software Development", IBM Technical Disclosure Bulletin, DA JHB, Vol 33 No. 6A, November 1990 pp Separation of user-interface from application, messages send to application from action bars (menus) or pop-up windows (pop-up menus). [FoksJ90a] Foks, Jack "Technology and open learning", Proc. OLNT'90, Australian Society for Educational Technology Overview of on-line teaching/instruction systems ca Describes Wang Freestyle as delayed communication/messaging system, point-to-point and point-to-multipoint (electronic annotation) [FoleyJD90a] Foley, James D.; vn Dam, Andries; Feiner, Steven K.; and Hughes, John F. "Computer Graphics: Principles and Practice, 1990 Edition", Addison-Wesley, 1995 (hardcopy book) Computer graphics reference, 1990 [FooteB89a] Foote, Brian "Class Warfare: Classes vs. Prototypes", Workshop on Objects without Classes, OOPSLA 8:26:33 PM]

173 ( ) History of Pen and Gesture Computing '90, New Orleans, LA Discussion of classless prototype-based languages, including Smalltalk and SELF. No description of development environments. [FrancikE89a] Francik, Ellen and Akagi, Kenichi "Designing a computer pencil and tablet for handwriting", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Design of WANG Freestyle tablet and stylus: taper, length, comparison of human factors of tablets. Gives samples of types of writing for tablet and stylus design. Freestyle stylus was engineered to look and feel like a pencil, with an "eraser" button (colored pink) on the rear end of the stylus. Tablet digitizer itself was electromagnetic, designed by Hitachi. [FujiwaraA89a] Fujiwara, A., Takenouchi, M., and Shimizu, M. "Character Recognition Apparatus", United States Patent 4,813,078, March 14, 1989, assigned to Matsushita Electric Industrial Company, Limited, Kadoma, Japan. Optical character recognition for typed Kana using direction codes / chain codes of outer perimeter to compare best matches of lengths of segments [FurukooriH89a] Furukoori, H. "Image Contour Detecting Apparatus", United States Patent 4,817,173, March 28, 1989, assigned to Kabushiki Kaisha Meidensha, Tokyo, Japan. Curve tracing [GTCO89a] GTCO Corporation "CalComp 9500 Digitizer Series: User's Guide", GTCO Corporation, Copyright 1989 CalComp/GTCO 9500 digitizer with Pressure Pen stylus: senses tip force, not pressure: electromagnetic / magnetic coupling, stylus is transmitter. Includes software to use tip sensor as switch, with adjustable force threshold. [GarwinRL89a] Garwin, R.L. and Levine, J.L. "Light-Sensitive Pen to Substitute for Finger in Laser Scanned Touch Screen", IBM Technical Disclosure Bulletin, vol 32 no 3B, pp (partial copy) High-resolution tablet using two laser scanners, photo-sensor on tip of stylus. Alternatively retro-reflector on stylus. [GaverWW89a] Gaver, W. W. "The SonicFinder: An interface that uses auditory icons", Human-Computer Interaction, Vol 4 No 1, 1989, pp Cited in Nielsen90: Add sound effects / auditory feedback to events in graphical user interface: trash sound when putting something into trash can, etc. Symbolic == beep, metaphorical == music, iconic == trash sound. Mentions utility for visually impaired, as well as other users. [GeiserG89a] Geiser, Georg "Handprinted Data Entry with a Touch-Sensitive Numeric Keypad", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Handwriting character recognition using a calculator's 5x5 (3x3 plus spaces between) keypad (similar to some Japanese calculator products): instead of double keying for alphabetics on a small calculator. Compare with Lucent/Microsoft touchpad calculator. Claims "very low" error rate: but unknown, does not say what the rate was. Shows number of variants/variations considered for handwriting recognition of single characters is NOT combinatorial. [GiordanoAG89a] Giordano, Albert G. and Disney, Christine "Desktop Dictionary of Information Systems Technology", Longman Group, 1989 (hardcopy book) Technical dictionary of computer terms 1989 [Go90] Go Corporation, Foster City, California "Go Corp. Gets IBM's Backing For 'Tablet PC'", PC Week, April 9, 1990, page 1 Go Corp. principals: Robert Carr (Aston-Tate), Jerry Kaplan (Lotus), Daniel Lewin (Next). Portable handwriting recognition product: PenPoint [GonzalezRC89a] Gonzalez, Rafael C. and Jerrera, Juan A. "Apparatus for Reading a License Plate", United States Patent 4,817,166, March 28, 1989, assigned to Perceptics Corporation, Knoxville, Tennessee. Features on license plates are edges, contours, height, width, topline and baseline, structural analysis, templates once the state (and therefor font) are known. [Goolsbey89] Goolsbey, Michael A., Jones, Eugene R., Micka, William F., and Robinson, Thomas, S. "Locating Individual Images in a Field for Recognition or the Like", United States Patent 4,817,186, March 28, 1989 Hardware to segment separate typed (handwritten?) characters for OCR recognition [Graphonomics89] International Graphonomics Society "Membership directory, Graphonomics", April 1989, c/o Department of Experimental Psychology, University of Nijmegen, PO Box 9104, 6500 HE Nijmegen, The Netherlands. Index of researchers in handwriting analysis, signature verification, and character recognition [GratkeNG89a] Gratke, Norman G. "Capacitance membrane switchcore with intertrace capacitive coupling and-or intratrace capacitive coupling", United States Patent 4,857,684 Capacitive (non-switch-contact) keyboard with driving and sensing rows and columns, multiple keys may be depressed simultaneously. multi-touch capable? [GreenM90] Green, Mark, and Jacob, Robert "SIGGRAPH '90 Workshop Report: Software Architectures and Metaphors for Non-WIMP User Interfaces", Computer Graphics, Vol 25 No 3, July :26:33 PM]

174 ( ) History of Pen and Gesture Computing UI/GUI design beyond pointing devices such as mouse and/or stylus digitizer: using gaze, sound, body motion: also gesture? [Greenstein89] Greenstetin, Joel S. and Baijal, Anish "An Investigation of Techniques for Occasional Numeric Data Entry", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp User-interface review of numeric keypad with mouse, multiple menus, sliders, clickers, simultaneous menus, used as substitute for a touchscreen: compare with Lucent/Microsoft materials. Reference to Quest Automation in 1970's as earlier handwriting recognition product. [GregoryWD89a] Gregory, William D. "Frequency dependent identification of materials", United States Patent 4,881,024, November 14, 1989 Capacitive sensing of material by measuring capacitance change at different frequencies, also change in phase. Electrode is segmented in 64-segment grid to measure different regions of object under test: allows profiles of classes of objects to check for internal structure / location of internal sub-objects. Also check Zimmerman 1995, Matias 2012 [GridPad89a] Grid Systems "Getting Started with GRiDForm", Grid Systems Corporation, September 1989 (hardcopy manual) Grid OS (on DOS) for tablet computer with handwriting recognition. Same binder as Program Development Manual [GridPad89b] Grid Systems "GRiDPAD Program Development Manual", Grid Systems Corporation, September 1989 (hardcopy manual) DOS-based tablet computer: Form definition graphical user interface. Allows for varying sizes of elements (radio buttons, etc.) Handwriting recognition context by subset / range/set of characters via HwrEnableChars. Returns list of possible characters with metric of certainty. P input from Pencept Penpad (but Grid has own handwriting recognition TSR?). Static buffers for tablet data, estimate 25 points/character (HwPInitialize) Same binder as Getting Started with GRiDForm [GridPad89c] Grid Systems "GRiDPAD A breakthrough in portable computing", Grid Systems Corporation, September 1989 Sales brochure with hardware specs (not software) for GridPad slate-style pen computer. [GridPad89d] Computergram International "Grid's GridPad handheld takes handprint data entry", Computergram International, October 2, 1989, no 1275 BYTE magazine gives award to GridPad product: compare 1985 award to PenPad. Additional GridPad information in file. GridPad 1910 ruggedized tablet computer, etc. One source cites GRiD Compass 1101 as "The first Clamshell Laptop", but it was predated by the DG/1 from Data General running MS/DOS. [GridPad89e] Computer Chronicles "Computer Chronicles: Portable Computers (1989)", Computer Chronicles video 1989 News presentation on portable computers, including touchscreen/pen computers: GridPad, Poqet PC, Dynabook 286, etc. Pre-Newton. Compare with GO / PenPoint. [GridPad89f] GridPad "GRiDPad Technical Specifications", Available at GridPad tablet computer technical specifications, includes information page from computer museum. Stylus attached. Resistive finger-touch touchscreen? [GridPad90a] BYTE "The BYTE awards (important products and technologies)", Byte magazine, January 1990, vol 15 no. 1, p GRiD System's GRiDPad, Lotus Magellan, Zenith MinisPort, Poqet PC, etc. [GridPad90b] Grid Corporation "GridPAD PadBase+ Training Course", course description, April 11, 1990, provided by Robert Artusy, R2Z Incorporated, Argonaut Way, Suite 319, Fremont, California GridPAD user-interface with stylus, training Correspondence includes list of features to be used for punctuation marks, handwriting prototype base forms and shapes [GrossbergS89a] Grossberg, Steven and Mingolla, Ennio "Neural Networks for Machine Vision", United States Patent 4,803,736, February 7, 1989, assigned to The Trustees of Boston University, Boston, Massachusetts. Boundary/edge/curve contour, feature contour, and object recognition in machine vision [GuiseDA89a] Guise, Dario A. "Efficient Frame Systems", Proc. EPIA 89 4th Portuguese Conference on Artificial Intelligence, pp Description of Garnet UIMS user interface management system, with color management. [GuoZ89a] Guo, Z. and Hall, R.W. "Parallel Thinning with Two-Subiteration Algorithms", CACM, Vol 32 No 3, March, 1989, pp Skeletonization/line-thinning using four compass directions, which preserves line connectedness [HaarslevV90a] Haarslev, Volker and Moeller, Ralf "A Framework for Visualizing Object-Oriented Systems", Proc. ECOOP/OOPSLA '90, October , pp Separating application and visualization in user-interface UIMS for automatic visualization of class hierarchies and graphs. Lisp/X-Windows, cites to visual programming [HaberS90a] Haber, Stuart and Stornetta, W. Scott "How to Time-Stamp a Digital Document", Proc. CRYPTO '90 8:26:33 PM]

175 ( ) History of Pen and Gesture Computing 10th Ann. Int'l. Cryptology Conf. on Advances in Cryptology, pp Digital time-stamping (certificate of date signed/existence of document) by time service signing hash of document and time value. Also forward-chaining of timestamps (hash of previous document) to prevent collusion of timestamp service to falsify record. Also in Journal of Cryptology Vol 3 no 2 pp , 1991 [HallRW89a] Hall, R.W. "Fast Parallel Thinning Algorithms: Parallel Speed and Connectivity Preservation", CACM, Vol 32 No 1, January 1989, pp Robert M Harlick is editor for many CA.C.M. OCR line-thinning articles OCR handwriting line thinning algorithm which does NOT eliminate diagonal lines [HarnadS90a] Harnad, S. "The symbol grounding problem", CNLS Conference on Emergent Computation, Los Alamos, May Also Physica D 42: pp Cited in MartinGL90 Relating abstract symbols to actual objects: relating symbols to symbols -- learning Chinese from a Chinese language dictionary alone. symbol system vs. semantics. [HauptmannAG89a] Hauptmann, Alexander G. "Speech and Gesture for Graphic Image Manipulation", Proc. CHI '89, May 1989, pp Wizard-of-Oz simulation of gesture/speech interface to computer, using three-dimensional 3D multiple hand and finger gestures. Mentions Rolky and Lee multi-touch input devices for prototypes. [HayashiY89a] Hayashi, Y., Sakata, M., Nakao, T., Ohno, T. and Ohhashi, S. "Alphanumeric character recognition using a connectionist model with the pocket algorithm", (abstract only), Proceedings of International Joint Conference on Neural Networks, Washington, D.C., June 18-22, 1989, Vol II p. 606 OCR for a single font using neural networks [HenryTR90] Henry, Tyson R.; Hudson, Scott E.; and Newell, Gary L. "Integrating Gesture and Snapping into a User Interface Toolkit", in in UIST'90 (1990), ACM, pp "X" cross-out gesture to delete; alphabetic gestures in a special "gesture area" on the tablet -- compare with Pencept PenPad? [HigginsCA90a] Higgins, Colin L. and Duckworth, R. James "The PAD (Ped And Display) - A Demonstrator for the Electronic Paper Project", Computer Processing of Handwriting, World Scientific Publishing, 1990, pp Electronic Paper Project EPP prototype of integrated tablet and display. Compare with IBM PLI Paper-Like Interface. Refers to parallax problems of touchscreen digitizer. Touchscreen (Scriptel: some parallax problems) used with Apple MacIntosh and MacWrite instead of mouse. [HildebrandTK89a] Hildebrand, Thoms K. and Stein, Norbert J. "Electronic book for reading textual information from an electronic medium / Elektronisches Buch zum Lesen literaturartiger Informationen aus einem elektronischen Medium", European Patent EP A2, published October 18, 1989 (in German) Electronic book system with flat display, touch-display-screen interface: touching a word or other element in text displays additional relevant information, boolean search expressions. [HorowitzSL89] Horowitz, Steven L. "Method and Apparatus for Generating Size and Orientation Invariant Shape Features", United States Patent 4,803,230, January 31, 1989, assigned to GTX Corporation, Phoenix, Arizona. Character recognition, OCR, statistical decision tree. OCR using features of six bounding rectangles at different angles, translated to direction (chain?) codes OCR using Fourier transforms, autocorrelations, moments, centroid of largest hole and center of character [HsiehAJ90a] Hsieh, A.-J., Kunk, S.-J., Shiau, S.-L., Kao, M.-C., and Chen, J.-W. "An Experimental System for Stroke-number Free On-line Chinese Character Recognition", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp Taiwanese efforts on both OCR and on-line Chinese character recognition: 5401 characters x 23 people x 10 samples each OCR, 5401 x 23 people (mixed) on-line data base collection. Later came by to visit Wang: part of five-year project, 9 people OCR, 5 on-line, first year is "find out everything there is to know" [Hubby89] Hubby, Laurence M. "Flat Panel Display with Integrated Digitizing Tablet", United States Patent 4,873,398, October 10, 1989 Optical digitizing tablet integrated with a flat (flat front surface CRT?) display. Light is emitted by the stylus, projected back through the same optical path used for projecting the image on the front surface (? ). Cites Scriptel. [HullJJ90a] Hull, Jonathan J., Commike, A., and Ho, T.-K. "Multiple Algorithms for Handwritten Character Recognition", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp OCR for handwritten Zip codes, address blocks. Description of IBM signature verification system at Federal Reserve [HurstJ89a] Hurst, Jan (chair); Mahoney, Michael S.; Taylor, Norman H.; Ross, Douglas T.; and Fano, Robert M. "Retrospectives: The Early Years in Computer Graphics at MIT, Lincoln Lab and Harvard", SIGGRAPH '90 Panel Proceedings, pp , (two parts) Light gun (light cannon) invented 1949 on Project Whirlwind. Origin of term "kludge". 8:26:33 PM]

176 ( ) History of Pen and Gesture Computing [IBM89a] Federal Computer Week "Smart Signatures at Fed", Federal Computer Week, May 22, 1989, Page 1 [IBM89b] IBM Corp. Research Division, press release "IBM Writing Interface May Replace Typing", spokesperson Gerald Present, September 1989, Description of IBM's "paperlike interface" user interface [IBM90a] IBM Corp. "Function-independent Approach to Driving Soft Keyboard", IBM Technical Disclosure Bulletin, Vol 33 No 4, September 1990 pp. 159-ff (partial copy) Soft/virtual keyboard using touchscreen on separate display: compare with Kaplow, Knowlton. Keyboard images and configuration can be changed for application, driver implementation. [IBM90b] IBM Corp. "Method of Providing On-Screen Keyboard for Touch-Sensitive Systems Without Affecting Application Programs", IBM Technical Disclosure Bulletin, Vol 33 No 5, October 1990 pp. 320-ff Pen-computing driver generates keyboard input from handwriting and gestures, work with existing applications. Compare with PenCept ca. 1986, CIC. [IBM90c] IBM Corp. "Pressure-Sensitive Icons", IBM Technical Disclosure Bulletin, Vol 33 No 13, June 1990 pp. 222-ff (partial copy) Pressure-sensitive touchscreen: show visual feedback on pressing by making the graphical image of a button look like it is being depressed. Mentions pressure for speed of scrolling. [IEEE90a] IEEE "IEEE Standard Computer Dictionary: Compilation of IEEE Standard Computer Glossaries", IEEE, 1990 IEEE Computer dictionary: standard reference. A bit hardware oriented. [IMCS90] International Machine Control Systems "MousePen product literature", January Vendels Circle, Paso Robles, California Stylus-shaped mouse with mechanical trackball: substitute for a tablet? [ITRI89] ITRI "Chinese Character Recognition", Wang Laboratories internal memorandum on discussions with Industrial Technology Research Institute (ITRI) in Taiwan, June 20, 1989 [ImpedovoS90a] Impedovo, S., Castellano, M., Pirlo, G., and Dimauro, G. "An Off-line Writer Identification System Based on a Syntactic Approach", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp Like signature verification, but using regular hand printing (a nothing paper, looks like they had to change horses from adaptive handwriting recognition in mid-stream) [IndustrialEE89] Industrial Electronic Engineers, Inc. "Pixie Graphic LCD Switch keyboard product literature", 1989, 7740 Lemona Ave, Van Nuys, CA 91409, User-interface item: keys with individual graphic LCD displays in the key top for relabeling the keys [InfoWorld90a] Krohn, Nico "Scenario Demonstrates Pen-Based Input System", InfoWorld, April 16, 1990 p. 8 Scenario Dynawriter tablet/display for PC with handwriting recognition: compare with CIC, PenCept, Paper- Like Interface of IBM, Sklarew. Mentions GridPad. [InfoWorld90b] Krohn, Nico "IBM Will License Go's Pen-Based OS for Laptops", InfoWorld, July 23, 1990 p. 1 IBM to license Penpoint operating system from GO, plans to test in 150 user sites. Portable to different architectures. Response on Pen Windows from Microsoft. [IshikawaY90a] Ishikawa, Y., and Kakajima, K. "A Real Time Connected Word Recognition System", 10th International Conference on Pattern Recognition, Atlantic City, New Jersey, June 16-21, 1990 Note on Chinese character recognition product internal to Wang Laboratories, but shown publicly. [JackelLD89a] Jackel, L.D. "VLSI Technology and Neural Network Chips", Tutorial Notes No 8, International Joint Conference on Neural Networks, Washington D.C., June 18-22, 1989 Contains examples of neural nets for handwritten numerals recognition in O.C.R. [JellinekHD90a] Jellinek, Herbert D. and Card, Stuart K. "Powermice and User Performance", Proc. CHI '90, April 1990, pp Study of powermice (mouse with velocity gain depending on speed of user input) for scrolling etc. Powermouse gain has no improvement on accuracy or pointing, more likely to be addressing problem of larger displays and smaller desk working area. [JohnsonJ89] Johnson, J., Roberts, T.L., Verplank, W., Smith, D.C., Irby, C.H., Beard, M., and Mackey, K. "The Xerox Star: A Retrospective", IEEE Computer, Vol 22 No 9, September, 1989, pp User-interface description of the development of the Xerox Star desktop paradigm, now called ViewPoint (originally on Alto computer) Xerox Star original desktop direct-manipulation user-interface intended only for casual users Desktop metaphor for user interface is intended for office applications an publishing, not software developments (tools based) Direct-manipulation desktop user interface (no processing on files per se) is orthogonal to tools-based user interface (UNIX/SunView Windows: any file can be operated on by any tool, even if not sensical) Clean user-interface lies more in Xerox Star details, not in features Mouse pointing input must be handled at low-level by system to prevent jerkiness, jumps, dead time as the system degrades: these (mouse or tablet or whatever) are eye-hand co-ordination devices One-button 8:26:33 PM]

177 ( ) History of Pen and Gesture Computing mouse (or tablet stylus) vs. two-button: fewer button confusion errors, but more increased selection errors Overlapped-windows cause users to spend time adjusting windows so they don't overlap: just use tiled windows to begin with (if the display is 17" large) Xerox Star: cites user-interface work by many others using graphical/sketch input, Xerox View Point [JohnstonJP89a] Johnston, James P. and Russell, Donald L. "Information Display and Entry Device", United States Patent 4,814,760, March 21, 1989, assigned to Wang Laboratories, Lowell, Massachusetts. Patent on integrated tablet and display (touchscreen) for a dynabook/electronic-ink. Tablet is capacitive, flexible/resilient layer surface on top can be pressed closer by stylus to capacitive sensor pad below, increasing capacitance. X/Y matrix grid of sensors. Also mentions detecting a change in resistance (impedance). Appears to measure each grid point separately (multi-touch?) [JonesD89a] Jones, D., Hapeshi, K., and Frankish, C. "Design guidelines for speech recognition interfaces", Applied Ergonomics, Vol 20 No 1, March 1989, pp Issues of user-interface design for task-specific, limited speech recognition: Syntax (context) has profound effects on speech variability, so design user-interface to minimize effects. Very important to collect samples (speech) in context for training: phonetic, semantic, and personal stress context affect variability. [JonesSV89a] Jones, Stacy V. "Bell System Develops Blackboard Which Displayes Notes Via Phone", Sarasota Herald-Tribue, Sunday November 19, 1978, p. 4-A Press announcement of Torok remote whiteboard, using standard phone lines. See US [Jourjine89] Jourjine, A.N. "Microprocessor Assemblies Forming Adaptive Neural Networks", United States Patent 4,809,193, February 28, 1989 Cellular automaton for neural nets, using radiated energy coupling between separate microprocessors [Jourjine90] Jourjine, Alexander "Variable capacity transmission medium", United States Patent 4,967,369 Neural net establishing correlation between signals by exchange of random codes [JoyceR90a] Joyce, Rick and Gupta, Gopal "Identity authentication based on keystroke latencies", CACM Vol 33 No 2, February 1990, pp Keyboard latency (biometric) to authenticate (verify identity of) user, handwritten signature shapes for signature verification. Cites to Rand 1980, but not to Saltzer [Katsurada89] Katsurada, Morihiro "Method of a System for Analyzing Characters", United States Patent 4,811,412, March 7, 1989, assigned to Shapr Kabushiki Kaisha, Osaka, Japan. Segment/parse character from right and left, see if they match, if they don't, try templates [Katt90] Katt, Spencer F. (pseudonym) "Microsoft ueber alles?", PC Week, April 16, 1990, page 182 "Rumor Central": Microsoft conference in Canada, microsoft starting development of Windows-based operating system for handwriting recognition, to compete with GRiD and Go Corporation, estimated release is 12 to 18 months [Kawakami90] Kawakami, Yasushi; Obata, Fukue; Makihara, Kayoko; Shiro, Yamada; Morimoto, Yoshinari et al "Data processing device with screen separated into line format area and ata display area", United States Patent 4,897,804, January 30, 1990 Text-input recognition area on tablet, data can be moved into regular display area [KayA89a] Kay, Alan "User Interface: A Personal View", in "multimedia: From Wagner to Virtual Reality", W. W. Norton and Company, Inc., 2001, pp Alan Kay essay on user interfaces. Includes historical notes on Dynabook and personal computers: no pen/tablet on Dynabook, but on earlier FLEX machine. [KellyW90] Kelly, William "Sparking Stylus for Acoustic Digitizer", United States Patent 4,891,474, January 2, 1990 SAC Science Accessories Corporation sparking stylus for acoustic/sonic tablet digitizer [Kensington89] Kensington Microware, Turbo "Mouse trackball product literature", Kensingon Microware, Park Avenue South, New York, New York Mouse/digitizer trackball replacement products [Khotanzad90] Khotanzad, Alireza and Hong, Yaw Hua "Rotation invariant image recognition using features selected via a systematic method", Pattern Recognition, Vol 23 No 10, October 1990, pp Character recognition using rotation-invariant features - Zernike moments (what about "E" and "W"?) Rotation-invariant OCR: 84% on numeric handprinted characters, 16,550 samples, 99% on "clean" alphabetics Numeric handprinting OCR samples from Recognition Equipment, Inc. Claims 84% correct recognition rate is very effective [KinnucanP89a] Kinnucan, P. "The Versatility of Graphics Tablets", Computer Graphics Review, Vol 4 No 7, July 1989, pp Simple-minded review article on digitizer technologies [KnightLW89a] Knight, Leland, W. and Retter, Dale "Datahand: Design, Potential Performance, and Improvements in the Computer Keyboard and Mouse", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Two handed mouse/keyboard user-interface combination, using the "Datahand" product. Keys are divided 8:26:33 PM]

178 ( ) History of Pen and Gesture Computing between the two hands, each of which is in a mouse-like movable housing. [Koezuka89] Koezuka, T.; Tsukahara, H.; Nakashima, M. "Pattern Matching Method and Apparatus", United States Patent 4,805,224, February 14, 1989, assigned to Fujitsu, Limited, Kawasaki, Japan. Video pattern matching recognition using two-dimensional extracted image from frame buffer [KreifeldtJG89a] Kreifeldt, J.G., Levine, S.L., and Iyengar, C. "Reduced Keyboard Designs using Disambiguation", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Stenotype-type user interface keyboard design: one key per finger, multiple characters per key [KreinPT90a] Krein, Philip T. and Meadows, R. David "The Electroquasistatics of the Capacitive Touch Panel", IEEE Trans. on Industry Application, Vol. 26, No 3, May/June 1990, pp Seems to be same as KreinPT88a: review of wavy-line distortion and inaccuracies in resistive/capacitive touch panels and tablets using resistive sheets. [KrzyzakA90a] Krzyzak, A., Dai, W., and Suen, C.Y. "Unconstrained Handwritten Character Classification Using Modified Backpropagation Model", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp [KuanCL89a] Kuan, C-C.L. and Srihari, S.N. "Analysis of handwritten text images using run-length based algorithms", IEEE Trans. Pattern Analysis and Machine Recognition, to appear, 1989 Cited in Taxt90 [KulpJ90a] Kulp, James and John "Personal communication: telephone conversation", April SuperScript Incorporated, Cambridge, Massachusetts, Working on a portable handwriting recognition product (with forms) John Kulp says they are a Symbolics spin-off: CD-ROM portable box for manuals. Did their own elastic matching: not super happy with it (using lots of context helps) [KunduA89a] Kundu, Amlan; He, Yang; and Bahl, Paramvir "Recognition of Handwritten Words: First and Second Order Hidden Markov Model Based Approach", Pattern Recognition, vol. 22, no. 3, p. 283, 1989 Cited by Marlin Eller, Microsoft Pen Computing group [KuoWC90a] Kuo, Wen-Cheh; Lin, Chia-Hui; Chou, Te-Chang; Lian, Yuan-Hsiung "Touch-control Computer Mouse", United States Patent 4,977,397, December 11, 1900 Small touchpad mounted in a mouse-like housing, to plug into a mouse port printed patent title miss-spells "mouse" as "house". [KurlanderD89a] Kurlander, David and Feiner, Steven "A Visual Language for Browsing, Undoing, and Redoing Graphical Interface Commands", Reprint from Visual Languages and Visual Programming, S.K. Chang ed.; Plenum Press, NY 1990, pp Graphical/visual programming language, but really a graphical editor user interface for "undo" showing command history in small graphical snapshots (panels) of what the image looked like [KurosawaY89a] Kurosawa, Y. "Pattern recognition apparatus with two-part reference storage", European Patent , Assigned to Toshiba Two-level OCR recognition using contours for one level, then detailed features for final resolution Performs choice of matching pattern from among number of candidate standards preselected using partial characteristic data. Describes tablet template, side-menu touch buttons for tablet (user-interface) Kurta's cordless stylus/pen for a digitizing tablet [Kurta89a] Kurta Corporation "Kurta IS/ONE, IS/ADB, and IS/TWO product description", Kurta Corporation, 3007 East Chambers Road, Phoenix, Arizona shows static templates GUI for picking commands in various applications [Kurta89b] Kurta Corporation "PageMaker, FreeHand, and Persuasion just got easier to use", Kurta Corporation, 3007 East Chambers Road, Phoenix, Arizona on-tablet templates with command menus, QuickKeys scripting language for keyboard shortcuts [Kuzunuki89] Kuzunuki, Soshiro, Shojima, Hiroshi, Miura, Masaki, Mori, Junko, and Mifune, Toshimi "Real Time Handwritten Character Input System", United States Patent 4,860,372, August 22, 1989, assigned to Hitachi, Limited, Tokyo, Japan. User interface on keyboard emulation with handwriting input: Ward85 was cited as prior art (check US Patent category 382,13). Handwriting user interface: use a Times-square circular buffer for writing in, because written character are bigger than typed font [LaLondeWR90a] LaLonde, Wilf R. and Pugh, John R. "Inside Smalltalk, Volume I", Prentice-Hall, 1990 Smalltalk object-based programming system, object-based programming language (prototype-based?), graphical user interface with windows, and IDE. Multiple classes of number data types: float, integer, and pure fraction. Transparent windows and background using a two-bit pixel map -- black/white, transparent/opaque, with transparent/white combination declared value undefined: compare to bit/mask layers in V25 hardware on Wang Freestyle The Guide. BitBlt support. View==Window. [LacyME89a] Lacy, Mark. E. "Artificial Laboratories", AI Magazine, Vol 10 No 2, 1989, pp :26:33 PM]

179 ( ) History of Pen and Gesture Computing Artificial reality environment, mentions virtual switches and virtual slide switches in graphical user-interface UI: input is by a DataGlove in three dimensions: refers to hand movements instead of gestures -- pinch gesture shown? [LandmeierWL89a] Landmeier, Waldo L. "Ink-on-Glass Digitizer Tablet and Method of Construction", United States Patent 4,873,399, October 10, 1989 Transparent touchscreen with spaced vertical conductive ink conductors on top surface of glass substrate, second set of horizontal conductors on bottom surface ITO? Transparent? Cites Rockwall patent on digitizer [LeCunY90a] Le Cun, Y.; Boxer, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., and Baird, H.S. "Constrained Neural Network for Unconstrained Handwritten Digit Recognition", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp In his notes, claimed neural nets can learn feature on their own: actually, his net structure, pixel resolution, etc. determined the class of features. OCR of handwritten zip codes: segmentation, "bad" samples already done. [LeeS89a] Lee, Seongwhan, and Kim, Jin H. "Unconstrained Seal Imprint Verification using Attributed Stroke Graph Matching", Pattern Recognition, Vol 22 No 6, 1989, pp Like signature verification, but for the stamps/seals used in oriental/chinese countries Spatial distributions of stroke skeletons: recognition independent of size/scale/magnification [LegaultR90a] Legault, R., Suen, C.Y., and Nadal, C. "Classification of Confusing Handwritten Numerals by Human Subjects", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp Similar to Shillman functional attribute experiments, using old zip-code samples Contains some hypothesis on distinguishing features for handwritten numerals and human recognition [LevineNA90a] Levine, Neil A. "Finger worn graphic interface device", United States Patent 4,954,817, September 4, 1990 Small touchpad (finger palette) worn on index finger, ring with bump for stylus dot worn on thumb. Touchpad can be worn while typing -- but not used concurrently with both hand typing. [LiHF89a] Li, H.F., Pao, Derek, and Jayakumar, R. "Improvements and Systolic Implementation of the Hough Transformation for Straight Line Segments", Pattern Recognition, Vol 22 No 6, 1989, pp Add contiguity check to Hough transform to find straight line features in a binary image: handles overlapping lines, short lines. [LiHF89b] Li, H.F., Jayakumar, R., and Youssef, M. "Parallel algorithms for recognizing handwritten characters using shape features", Pattern Recognition, Vol 22 No 6, 1989, pp Static/OCR handwriting recognition: features are local shape, start and end points, edge types, and join relations in contours (digits only!) OCR for handwritten digits: 120 samples per digit Handwriting OCR: twelve relations for contours: compare to combinatorial explosion in Ward88? chain codes via systolic array [Lien89] Lien, Yih-Liang L. "Apparatus and Method for Vectorization of Incoming Scanned Image Data", United States Patent 4,817,187, March 28, 1989, assigned to GTX Corporation, Phoenix, Arizona. Curve-tracing, curve-follower, scanner algorithm for detecting lines in OCR or image analysis. [LinWC90a] Lin, Wen C. "Handbook of digital System Design, 2nd Edition", CRC Press, 1990 (hardcopy book) Upper-level textbook on digital circuits and systems. More focus on physical device properties (analog properties / switching frequency / etc.) [LinnJ90a] Linn, John "Practical Authentication for Distributed Computing", Proc IEEE Symp. on Research in Security and Privacy, Los Alamitos, California IEEE Computer Society, 1990 pp Overview of DSSA Distributed System Security Architecture (DEC Digital Equipment Corporation): systems are able to trust each other because of PKI authentication of identity and integrity of messages. Identity certificates from a CA, also dynamic binding certificates between two parties for delegation of authority: compare with SSL protocol. Certificates used to authenticate software programs / software images during boot/loading. [LintonMA89a] Linton, Mark A.; Vlissides, John M.; and Calder, Paul R. "Composing User Interfaces with InterViews", IEEE Computer, February 1989, pp UIMS (Garnet/Interactors) in C++, colors and appearance of UI defined in run-time calls to C++ classes, including color management, model is based on X toolkit. Multiple UIs for same application, UIs need not be entirely graphical. [LoganJD89a] Logan, Jame D. and Litvin, Yury "Touch Screen Computer-Operated Video Display Process and Apparatus", United States Patent 4,821,029, April 11, 1989 Touch screen UI: select a button or area, information is copied to a different window or area on the screen. [Logitech89] Logitech Corporation "LogiMouse C7 Technical Reference Manual", Logitech Corporation, :26:33 PM]

180 ( ) History of Pen and Gesture Computing Electronic digitizer tablet sold as a mouse replacement: emulates Microsoft Mouse data format, other tablets [LuS90a] Lu, S.W. and Suen, C.Y. "Hierarchical Attributed Graph Representation and Recognition of Handwritten Chinese Characters", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp (abstract only) Chinese/Kanji OCR [MCC90] MCC Microelectronics and Computer Technology Corporation "Interactive Tablets", description of work in progress, CHI '90 Conference on Human Factors in Computing Systems, Seattle Washington, April 2-3, Available from Tom Parish, MCC, PO Box , Austin Texas 78720, General description of neural net character recognition, user-interface for handwriting at MCC MCC staff on handwriting/gesture user interface: Tom Parish (general), Jim Pittman and David Chapman (hardware), Gale Martin (recognition), Kent Wittenburg (interpretation), Bill Gooch (Prototypes) Working on mathematical expression user-interface stroke segmentation [MacTutor90a] MacTutor "MacTutor: The Macintosh Programming Journal", MacTutor: The Macintosh Programming Journal, Vol 6. No. 3, March 1990 (hardcopy book) General issue on programming development and environments for MacIntosh, commercial IDEs, Hypercard. Build environments similar to PenApps/ Visual Basic: Prototyper 2.0 from SmethersBarnes draws and generates code: HyperBasic from Teknosys, compiler linker and resource mover for XCMDs and XFCNs (HyperCard and Hypertalk extension); MacInterface from Holder Egand and Company: ToolKit HyperCard script editor; Classy Object Factor with Application Generator; Prograph 1.2 OOP programming CASE tool; Serius Desktop programming Serius79 for graphical custom applications. [MacTutor90b] MacTutor "MacTutor: The Macintosh Programming Journal", MacTutor: The Macintosh Programming Journal, Vol 6. No. 4, April 1990 (hardcopy book) p. 84, TooLKit MenuMaker by Shared Intelligence for Hypercard development, p. 50 HyperBasic for HyperCard, p. 33 Prograph by TGS Systems, p. 13 ScriptEdit for HyperCard. Build environments similar to PenApps/ Visual Basic: Prototyper 2.0 from SmethersBarnes draws and generates code: HyperBasic from Teknosys, compiler linker and resource mover for XCMDs and XFCNs (HyperCard and Hypertalk extension); MacInterface from Holder Egand and Company: ToolKit HyperCard script editor; Classy Object Factor with Application Generator; Prograph 1.2 OOP programming CASE tool; Serius Desktop programming Serius79 for graphical custom applications. [MackR89a] Mack, Robert, and Lang, Kathy "A Benchmark Comparison of Mouse and Touch Interface Techniques for an Intelligent Workstation Windowing Environment", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Replace mouse with touchscreen digitizer tablet on screen to Windows 2.0 by using new driver. Refers to double-click/double-tap gesture problems of mouse versus keyboard, but overall positive. Suggests delaybefore-select as alternative to select-on-lift when using touchscreen with stylus, so that user has some time to "fine-tune" click/tap/select position, also refers to maintain-contact to select followed by drag. Dwell/press-and-hold -- maybe combineable: click-hold-drag / maintain-contact-and-drag is ordinary drag. Stylus faster than keyboard for pointing. [MahachKR89a] Mahach, Karen Renee "A Comparison of Computer Input Devices: Linus Pen, Mouse, Cursor Keys and Keyboard", "Linus Pen, Mouse, Cursor Keys, and Keyboard", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Linus electronic ink tablet/display had low accuracy: typing faster than writing: points out that a handwriting user-interface is not really just handwriting, but a lot of menu-picking for data entry, which needs to be studied Looks like a good researcher to contact on electronic ink studies [MaltzIJ89a] Maltz, Ivan J. "Video Compositing Using a Software Linear Layer", United States Patent 4,827,253, May 2, 1989 Overlay foreground image (on a touch display, etc.) by having a separate blending-factor palette look-up table given blending factor (alpha-blending) for particular pixels, separate from the images being overlaid. [ManessWL89a] Maness, William L.; Golden, Robert F.; Benjamin, Michael H.; and Podoloff, Robert M. "Pressure and contact sensor system for measuring dental occlusion", United States Patent 4,856,993, August 15, 1989 Pressure-sensing X/Y grid of conductors separated by pressure-sensitive resistive ink material only over the row electrodes (not a sheet or film), used for dental occlusion check. Gives formula for resistive and conductive inks. Not clear how it distinguishes "ghost" inputs. [Marshall89] Marshall, William W. and McWhortor, William F. "Method and Apparatus for Pattern Recognition", United States Patent 4,817,176, March 28, 1989, assigned to William F. McWhortor, Fountain Hills, Arizona. Character recognition from OCR using Fourier transforms, returning confidence percentage goodness based on second-best match. [MartinGL89a] Martin, Gale L. "The utility of speech input in user-computer interfaces", International Journal of 8:26:33 PM]

181 ( ) History of Pen and Gesture Computing Man-Machine Studies, Vol 30 No 4, April 1989, pp Points out that user-interface questions for speech recognition are hard to answer, because so many interaction modalities are involved Points out indirectly (for Votan speech recognition) that the "perfect" recognition of 100% went down over time after training. Grand review of speech recognition in userinterfaces Speech recognition is faster than typing, and it has the advantage of being an additional channel of input besides the hands (haptic?) For a simplified set of commands, speech recognition and mouse clicks equally efficient Report by group at MCC Microelectronics and Computer Technology Corporation on their handwriting/speech recognition system (MAGIC) Cites a large number of studies that show speech recognition is not as good as keyboard input (!!), although this paper says speech is better [MartinGL90a] Martin, Gale L., Leow, Wee Kheng, and Pittman, James A. "Function Complexity Effects on Backpropagation Learning", MCC Technical Report ACT-HI , Microelectronics and Computer Technology Corporation, Human Interface Laboratory, 3500 West Balcones Center Drive, Austin, Texas 78758, January 1990 Neural network report on on-line handwriting recognition Data collection of 8,600 upper-case letters from (from NCR for digits) 110 people on an integrated tablet/display, 200 points/second 1000 points/inch Training sets of 500,2500,6261 sample from different people States one needs tens of thousands of samples of handwriting data base to train a system, more than an adaptable neural net Neural nets need to be designed to detect human features, otherwise not much good [MartinGL90b] Martin, Gale L., and Pittman, James A. "Recognizing Hand-Printed Letters and Digits Using Backpropagation Learning", MCC Non-confidential report, to appear in Advances in Neural Information Processing, D.S. Touretzky, editor, Morgan Kaufman, 1990 Cites NCR corporation data-base of hand-printed on-line character shape collection 8,600 on-line handwriting samples from 110 people, 200 points/second at 1000 points/inch: images converted to blurred 15x24 grayscale image for static character recognition Training set of 500 to 6,300 samples, 18 people, separate from test data: human handwriting recognition error rate was 3.4% (compare Neisser60) Gives table of exact trade-offs between reject rate and error rate on handwriting recognition, for digits and letters separately Estimate of number of training samples needed for adaptive recognizer / neural network Credits at MCC/NCR to Dave Rumelhart, Joyce Conner, Janet Kilgore, and Kay Bauer for collection [MartinGL90c] Martin, Gale; Pittman, James; Wittenburg, Kent; Cohen, Richard; and Parish, Tom "Sign Here, Please", BYTE Magazine, July 1990, pp Overview article on handwriting/gesture and voice recognition, especially parsing/chunking problems, neural net trainable handwriting recognizer at MCC: mentions DragonDictate, Freestyle, VoiceBox from Voice Recognition Technologies, Interactive Worksurface Project (tabletop? / blackboard metaphor?) at MCC, other voice recognition companies and products. [MarzinkewitschR90a] Marzinkewitsch, Reiner "Ein Arbeitsplatz zum rechnerunterstützten handschriftlichen Rechnen mit mathematischen Formeln", Dr. Nat. Dissertation, Uni. des Saarlandes, Saarbrücken, 1990 Ph.D. thesis on pen-computing/tablet input of mathematical formulas, editing, on graphics tablet (complaints about the stylus/griffel). (in German). Sections on 2D parsing and segmentation. Discussion of context-free 2D grammars. Cites to Anderson 70, not to Martin/Kaplow. [MathewsMV90a] Mathews, Max V. "Three Dimensional Baton and Gesture Sensor", United States Patent 4,980,519, December 25, 1990 See Radio Drum: Electrostatic grid of sensor pads, sense two or more styli (drumsticks) in X, Y and Z by capacitive coupling. Pads have segmented shape so that shielding of one layer by another is minimized. Tapered conductors: charge ratio? Baton striking surface (touching) is inferred/predicted from Z motion. [MatthewsHG89a] Matthews, Henry G.; Zalenski, Thomas; Barbetti, Jamie L.; and Mletzko, Al "Method and apparatus for calibration of a charge ratio digitizer", United States Patent 4,831,566, May 16, 1989 Self-Calibration for a charge ratio (or really any capacitive) digitizer: phantom/simulated cursor/stylus electrodes at known positions, check for reported position of the phantom cursors and calculate correction factors. [MaulsbyDL89a] Maulsby, David L.; Witten, Ian H; and Kittlitz, Kenneth A. "Metamouse: Specifying Graphical Procedures by Example", ACM Computer Graphics, Vol 23 No 3, July 1989, pp Programming by example: deduce macro operations for graphical actions by applying constraints to observed operations in GUI. Must infer variable (such as row indices) [McArthurD90a] McArthur, David; and Stasz, Cathleen "An Intelligent Tutor for Basic Algebra", RAND Report R NSF Prepared for The National Science Foundation, August 1990 Two-dimensional mathematical input: not clear on the input means for user interface, perhaps on-line handwriting recognition [McAvinneyP90a] McAvinney, Paul and Rubine, Dean H. "Videoharp", United States Patent 4.968,877, November 6, 1990 Electronic VideoHarp musical instrument using optical sensing of multiple fingers (multi-touch) and gestures: either harp-like (strum) or piano-like (projected keyboard). Optical sensing on two sides (either 8:26:33 PM]

182 ( ) History of Pen and Gesture Computing side) of transparent instrument, such that instrument can be played with both hands like a harp, or mounted horizontally like a piano. MIDI output. to a synthesizer. Position of finger along a ray detected by shadowing. See also RubineD88a [McDonaldJE89a] McDonald, James E. and Stone, Jimmy D. "Parallax induced pointing error avoidance method and means for systems using touch screen overlays", United States Statutory Invention Registration H716, December 5, 1989 Put margin around active touch areas (button) for touchscreen on CRT display for safer targeting / pointing error. Margin sized to match parallax error / accuracy. [MeadowsRD90a] Meadows, Robert D. "Integrated Liquid Crystal Panel and Optical Touch Panel", United States Patent 4,916,308, April 10, 1990 Optical touch screen with high resolution equal to LCD pixels, small angled mirrors reflect light from outermost row/column of LCD display to be the light beams across the display. Stylus for high resolution. [MeeksML90a] Meeks, M.L., and Kuklinski, T.T. "Measurement of Dynamic Digitizer Performance", to be published in Computer Processing of Handwriting, Leedham and Plamandon, editors, World Scientific, Singapore, 1990 Report on tablet characteristics and instrumentation for measuring handwriting Describes test bed for calibrating tablet digitizers with an X/Y table to correct for *fixed* positional errors -- not tilt or velocity related accuracy problems? [Microslate89] Microslate "press release", Microslate, 1989 Rumored touchscreen portable product: handwriting recognition? [Microsoft90a] Microsoft Corporation, announcement of "Windows/H", Product development, PC Week "Spencer F. Katt" column, April , p. 182 Microsoft (Lloyd Frink and Marlin Eller) on Windows/H PenWindows user-interface for handwriting recognition: using "Anfa" or "Infa" company [Microsoft90b] Microsoft Corporation "Windows of Opportunity: Microsoft OEM Briefing 1990", Microsoft, Bellevue WA, October 1-3, 1990 Included information "Information at your Fingertips". Windows/H Windows for Pen Computing: rotation for Pen Windows computer screen for vertical or horizontal pad of paper orientation. Pen Windows will allow the user to rotate the screen. Stylus senses rotation of stylus barrel and pressure (tip force), proximity above screen. Side button to distinguish between handwriting, gestures, mouse-like pointing actions. Virtual keyboard / synthetic keyboard with writing window. Gestures, writing input with position. Drag, select, bracket gestures. fetched from [Microtouch89a] Microtouch "Microtouch unmouse product file", (various) Microtouch.com File of references regarding PC UnMouse from microtouch: resistive-film small touchpad/touchscreen panel on a mechanical switch, user can drag finger across surface of touchscreen to simulate proximity information or mouse motion, user presses on entire surface to close switch to for mouse click / touch-lift. Usually operated in relative-motion mode, not absolute digitizer tablet mode. Available for DOS/Windows PC and for MacIntosh. [Microtouch90a] (no author) "ScreenWrite Touch Screen Performance Specifications", Available from Microtouch Corporation, Microtouch.com Digitizer touch-screen/tablet, shows calibration/linearization template for adjusting for regional position errors -- rectangular grid with alternating rows/columns offset. [MillerBP90a] Miller, Barton P.; Fredriksen, Louis; and So, Bryan "An empirical study of the reliability of UNIX utilities", Comm. ACM, Vol 33 No 12, Dec. 1990, pp Fuzz-testing: random keyboard input to Unix utilities (very mature programs) causes most to fail, because few programmers check plausibility of user input, and thus fail on unanticipated input. [MingJ90] Ming, Ji "The Statistical Information Formulation for Noisy Speech Recognition", 10th International Conference on Pattern Recognition, Atlantic City, New Jersey, June 16-21, 1990 [MinskyM90a] Minsky, Margaret; Ming, Ouh-young; Steele, Oliver; Brooks, Frederick P. Jr.; and Behensky, Max "Feeling and seeing: issues in force display", Proc Symp. on Interactive 3D Graphics: SIGGRAPH, Vol 24 No 2, mar. 1990, pp Haptic feedback to give the illusion of a surface requires surface, texture, temperature, softness, etc. Software synthesizes, also viscosity. Special note on stability: analog integration required with digital control, raising joystick sampling from 30Hz to 1000Hz was not enough, wanted to achieve similar stability to human arm (which has 200msec response time, but is still stable) [Mira89] Mira Imaging, Inc. "Hyperspace (tm) product literature", Logan Avenue, Salt Lake City, Utah, D digitizer software: works with Science Accessories, IsoTrak (Polhemus), Cyberware laser scanner, Topcon. [MoellerR90a] Moeller, Ralf "Entwicklung von Visualiserungswerkzeugen in objektorientierten Systemen unter Verwendung von KI-Programmiermethoden", Universität Hamburg, Technical Report B-149, :26:33 PM]

183 ( ) History of Pen and Gesture Computing (partial copy) 1989 Doctoral thesis: describes LabView and other visual programming languages, page 13 shows virtual slide switch element [Momenta89] Momenta Inc "press release", Milpitas California, October 19, 1989 Rumored handwriting recognition portable product from Momenta [Momenta90] Display Momenta Corporation "What's New in Laptops and Pen Computing", Bill Lempesis of Dataquest, Flat Panel Display News, Vol 1 No 3, May/June 1990 Rumors of Go Corporation, Grid, Momenta: contains a P.R. blurb from Momenta on "four waves of personal computing user-interfaces", leading up to verbal/voice recognition. [MoogRA90a] Moog, Robert A. and Rhea, Thomas L. "Evolution of the Keyboard Interface: The Boesendorfer 290 SE Recording Piano and the Moog Multiply-Touch-Sensitive Keyboards", Computer Music Journal, Vol 14 NO 2, Summer 1990, pp Moog Multiply Touch-Sensitive keyboard (MTS keyboard) with resistive-sheet X/Y sensing, conductive rubber capacitive sensing of Z/force/pressure, velocity (photographs). Similar Boesendorfer 290 (modern) piano recorder. [MoriR90a] Mori, Ryoichi and Kawahara, Masaji "Superdistribution: The Concept and the Architecture", Trans of IEICE, Vol E73, No 7 July Special Issue on Cryptography and Information Security DRM document distribution system, software usage fees and royalties, permission. Uses trusted software secure hardware module (S-box) for real-time-clock for timestamps and storing private keys, hardware detection of hardware tampering. Electronic file contains additional information. See ContentGuard Google. [MoriY89a] Mori, Y. and Yokosawa, K. "Neural networks that learn to discriminate similar kanji characters", in Advances in neural information processing systems I, D.S. Touretzky, editor, Morgan Kaufman, 1989 cited in MartinGL90 [MyersBA89a] Myers, Brad A. "User-Interface Tools: Introduction and Survey", IEEE Software, Jan 1989, Vol 6 No 1, pp In Garnet system papers: refers to need for one application to have many user interfaces, separation (as a design question?) of user-interface an application. [MyersBA89b] Myers, Brad A.; Guise, Roger B; Zanden, Brad Vander; Kosbie, David; Marchal, Philippe; Pervin, Ed; Kolojejchick, John A. "The Garnet Toolkit Reference Manuals: Support for Highly-Interactive, Graphical User Interfaces in Lisp", Carnegie-Mellon University report CMU-CS , November 1989 In Garnet system papers: KR object model, updates to prototypes (using Garnet interface builder) automatically reflected in all instances on screen. Circular gestures for rotating objects. Dynamic themes? "A high-level tool, such as the Garnet interface builder, can display a prototype on the screen, and allow the user to edit it. These edits are then automatically reflected in all instances of that prototype. For example, the designer might be changing the standard look-and-feel of the menu prototype, and immediately all menus in the system will change accordingly." [MyersBA89c] Myers, Brad A.; Zanden, Brad Vander; and Dannenberg, Roger B. "Creating Graphical Interactive Application Objects by Demonstration", Proc. SIGGRAPH 89, pp Same title as Myer's '86 paper? In Garnet system papers: Lapidary user interface tool. Supports different styles, examples are MacIntosh-like and OpenLook-like. Also shapes of objects can be changed: square to circle, etc. Dynamic themes. Page 97: graphical replace mechanism allow wholescale replacement or modification of graphical objects at run/design time: e.g. round buttons to checkboxes, etc. [MyersBA89d] Myers, Brad A. "Encapsulating Interactive Behaviors", Proc. CHI '89, May 1989, pp In Garnet system papers: Encapsulate interactive behaviors into "Interactor" class objects, separate from GUI-element objects. New interactors can be defined for new kinds of input, such as gesture and handwriting recognition. Interactors can handle two-handed (multi-touch) input, e.g. mouse and touch tablet at same time. [MyersBA90a] Myers, B.A., Guise, D., Dannenberg, R.B., Zanden, B.V., Kosbie, D., Pervin, E., Mickish, A. and Marchal, P. "Garnet: Comprehensive support for graphical, highly-interactive user interfaces: The Garnet user interface development environment", IEEE Computer, Vol 23 No 11, Nov. 1990, pp ; also in Baecker95a Toward the Year 2000 Garnet system: Constraint-based graphical editor for constructing GUIs: visual programming language for rapid UI prototypes. Look and feel independent GUI gadgets/objects include floating buttons, number sliders, gauges, text fields, and scroll bars. When designer changes standard look and feel, all menus in system immediately change accordingly: example is changes to shapes, font, color of buttons, slide bars. Uses Opal (language) as graphical object system. Defines "slots" as fields or instance variables, using prototype inheritance. Pop-up virtual keyboard? SIP soft input panel? dynamic themes? [MyersBA90b] Myers, Brad A. "A New Model for Handling Input", ACM Trans. on Inf. Systems, Nol 8 No 3, July 1990, pp Garnet system: UIMS/IDE supporting arbitrary looks and feels, programmer can control both appearance and feedback behavior of UI elements such as menus. UI defined independent of the graphics and 8:26:33 PM]

184 ( ) History of Pen and Gesture Computing application program via Interactors. Interactors can be attached to graphics on the fly, even while interface is running. Includes definition of "slots" as accessible dynamic parameters to an object. Shows dynamically-changeable themes? [MyersBA90c] Myers, Brad A. "Demonstrational interfaces: A step beyond direct manipulation", Carnegie Mellon University Research Showcase, Computer Science Department, 1998 User interface: example values or example operations displayed while user is using user interface. If includes inferencing of user intent, called intelligent user interface. Example-based programming, direct manipulation. [NCR90] NCR Corporation, press release "Workstation Sales Systems to be Bought from NCR", May 15, 1990 Press-release on NCR signature capture / handwriting electronic ink system [NCSC89a] National Computer Security Center, Trusted Unix Working Group (TRUSIX) Fort Meade, Md. "Rationale for Selection Access Control List Features for the Unix (R) System", NCSC-TG-020-A, Library No. S-232,508 Study/design report on ACL access control list features for Unix: multiple permissions beyond read/write/execute access mode, multiple entries for users and for groups. Additional modes/permissions/rights can be defined. Specifies syntax of ACL entries. [Nagasawa89] Nagasawa, H., Ito, K., Horii, S., and Tanoshima, K. "Optical Character Reader Apparatus", United States Patent 4,799,271, January 17, 1989, assigned to Oki Electric Company, Limited, Tokyo, Japan. OCR character recognition matching entire words against a dictionary [Nakamura89] Nakamura, Y., Ooi, K., and Yura, K. "Hand-held data input apparatus", United States Patent 4,866,646, September 12, 1989, assigned to Toshiba. Patent on external handwriting recognition product with integrated tablet/display to attach to keyboard port, and voice output User-interface for electronic ink, editing documents Recognizes handwritten character data input through force/pressure sensitive tablet and sequentially stores result of character data [NakaneK90a] Nakane, Neiichi; Kuwabara, Naoya Ikeda; Koreeda, Hiroyuki; Aotsu, Hiroaki; Kawase Masaki; Tatsuno, Yujiro; Nonaka, Naomichi; and Suzuki, Kazunarai "Multi-window display control system", United States Patent 4,954,818, September 4, 1990 Displaying data in multiple windows [NakataniM89a] Nakatani, Munehiro "Image Processing Apparatus", United States Patent 4,817,174, March 28, 1989, assigned to Minolta Camera Kabushiki Kaisha, Osaka, Japan. Edge contour/curve detection for machine vision: for focusing cameras? [NegroponteN89a] Negroponte, Nicholas "A Personal Perspective: An Iconoclastic View Beyond the Desktop Metaphor", International Journal of Human-Computer Interaction, Vol 1 No 1, 1989, pp User-interface: a desktop is NOT complete human communication: hand gestures, winks (body gestures), tone of voice, context of the situation, etc. [Neurogen89] CAP International, Inc. "Neurogen, Inc.: Reading Numbers with Neural Networks", Market Report, October 2, Principals: Kupfersmith. One Longwater Circle, Norwell, Massachusetts, 02061, Neurogen claims 99% correct OCR recognition on handwritten numerals, but with 30% rejection rate Report on Neurogen, other handwriting OCR recognition companies for numerals : Nestor, Hect-Nielson Neurocomputer, Siemens, REI/ Recognition Equipment Incorporated, Scan-Optics, Elsag (Italy), Recognitive Sciences, Calera, Datacopy Accutext, Caere OmniPage, Calera TrueScan [NewellA90a] Newell, Allen and Fahlman, Scott E. "Information Processing Research report WL-TR ", Information Processing Research report WL-TR , CMU, Published 1992, Final Report for Period July July 1990 Section Research in Uniform Workstation Interfaces: Garnet Project replaced Dante project. LISP/X11 UIMS system for allowing users to construct widget GUIs using constraints. [NichollsHR89a] Nicholls, Howard R. and Lee, Mark H. "A Survey of Robot Tactile Sensing Technology", International Journal of Robotics Research, Vol 8, No 3, June 1989, pp Tactile-array touchpad sensors for robotics: 2D sensors include through thickness of resistive rubber, laterally across rubber between adjacent electrodes; matrix mats of carbon fibers (nanotubes?); capacitive grid; compliant elastomeric membrane pressed on surface affecting internal reflections; Separate section on recognizing shapes from static patterns on surface array (multitouch) [NielsenJ89a] Nielsen, Jakob "CHI '89 Trip Report", Comments on demonstration of the Paper-Like Interface with handwriting recognition by Cathy Wolf et al from IBM, with a "formula editor" (spreadsheet/mathematical notation) application. Also a mention of gestural interaction in an application toolkit for prototyping user interfaces from Mark Rosenstein of MCC Human Interface Tool Set and "silicon paper" (electronic paper). Several paragraphs on Wang Freestyle, described as an application with the ability to integrate graphics, spoken comments, and animated writing (miss-called "gestures"). [NielsenJ90a] Nielsen, Jakob "Traditional Dialogue Design Applied to Modern User Interfaces", Communications of the ACM, Vol 33 No 10, October 1990, pp :26:33 PM]

185 ( ) History of Pen and Gesture Computing Review of Wang Freestyle, other pen and non-pen UIs: shows postage stamp feature of Freestyle for documents. Says scrolling of text is bad in UI design. [NolanPR89a] Nolan, Peter R. "Designing Screen Icons: Ranking and Matching Studies", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Suggested icons for user-interface: symmetric confusions, "confusion matrix" of preferred selections [NormileD90a] Normile, Dennis and Johnson, J.T. "Computers Without Keys", Popular Science, August 1990, pp Soni Ai Note PalmTop with handwriting recognition and electronic ink drawing, GridPad, SuperScript, GO Corp./PenPoint (not yet released). Typing plate with boxes for handwriting recognition, with send/transfer button at end. Refers to template-matching recognizer. [Numonics89a] Numonics Corporation "Numonics Series 2300 Microprocessor-Based Digitizer Tablet", Numonics Corporation, 1989 Electromagnetic digitizer, includes commands for setting scaling, offset, origin and rotation/tilt of digitizer coordinate system. [NunezLM89a] Nunez, Linda Mensinger "professional resume", for work in on-line handwriting recognition, 1989 Resume for former employee of Nestor, Incorporated, worked on Japanese on-line handwriting recognition in 1987 [ON90] ON Technology Inc. "ON Technology, press release", ON Technology, April 24, 1990 Peter B. Miller to start handwriting recognition-base systems software company for Slate/tablet (touchscreen) computers. [OkamotoE90a] Okamoto, E. and Masumoto, H. "ID-based authentication system for computer virus protection", Electronic Letters, Vol 26 No 15, July 19, 1990 PKI signature with certified key (certificate) on software to verify it does not contain a virus [OoiK89] Ooi, K.; Sasaki, H.; and Ariyoshi, S. "Character recognition system", European Patent (number incorrect?) July 26, 1989, assigned to Toshiba (abstract only) Includes device for obtaining coincidence between recognition character candidates and character patterns stored in dictionary memory [OssnerJ90a] Ossner, J. "Transnational symbols: The rule of pictograms and models in the learning process", in "Designing User Interfaces for International Use", J. Nielsen, Ed., Elsevier Science Publishing, Amsterdam, 1990, pp Cited in Nielsen90 [PCLaptop89a] Patrick, Mark and Sachs, George "Richman Maximizer PIM Contact Manager", PC Laptop magazine, October 1989, page 41 Announcement for PIM/Contact manager: MS-DOS. [PCLaptop89b] Patchett, Craig "Touchscreen and CD-ROM: Is the Scenario Dynabook too good to be true?", PC Laptop magazine, October 1989, pp Touchscreen DOS PC with no keyboard: main application is as electronic book with built-in data CD-ROM drive. SAW surface acoustic wave technology, pressure/force sensitive. Three-point calibration (X + Y, no tilt/rotation?) [PastorJ89a] Pastor, Jose "Reliable Document Authentication System", United States Patent 4,853,961, August 1, 1989 Authenticate documents (or, authenticate postage meter stamp) using public-key cryptography by third party. Authenticating document with public/private key shows source authentication of document, also discloses key for decrypting document. Documents can be authenticated without being read. (Seem a bit pre-certificate.) [PatrickM89a] Patrick, Mark and Sachs, George "X11 Input Extension Protocol Specification", M.I.T. X Consortium - Technical Review - X11R4 Public Release, December 1989 Extension to X graphical system standard to support devices beyond system keyboard and system pointer (mouse): event input includes Key, Button, Valuator data in motion events, proximity events, focus, feedbacks [PenPoint90a] GO Corporation "Using the GO Computer", GO Corporation internal document: September 26, 1990 Description of user interface of PenPoint operating system, Selectable screen orientation landscape/portrait. Orientation/rotation PDF page 187 Choosing a screen layout. [PentlandA90a] Pentland, A. and Kuo, J. "The Artist at the Interface", 10th International Conference on Pattern Recognition, Atlantic City, New Jersey, June 16-21, 1990 [PeppersNA89a] Peppers, N.A., Young, J.R., Pierce, G.A., Nishi, H., and Katsuki, K. "Apparatus for Preprocessing of Character Recognition", United States Patent 4,809,344, February 28, 1989, assigned to Nippon Sheet Glass Company, Limited, Osaka, Japan. Segmentation in OCR scanner input for paragraphs, words picture, and individual letters, also doing character normalization. 8:26:33 PM]

186 ( ) History of Pen and Gesture Computing [PeppersNA89b] Peppers, N.A., Young, J.R., Nishi, H., and Katsuki, K. "Page Segmenter", United States Patent 4,817,169, March 28, 1989, assigned to Nippon Sheet Glass Company, Limited, Osaka, Japan (abstract only). Segmenting a document image into parts "with a single feature" as pre-processing before pattern recognition [PerkinsR89a] Perkins, Ronald; Blatt, Louis A.; Workman, Daniel; Ehrlich, Susan F. "Iterative Tutorial Design in the Product Development Cycle", Proc. Human Factors and Ergonomics Society Annual Meeting, 1989, pp (partial copy) Report on tutorial for Wang Freestyle, but indirectly a human factors report on usability of Freestyle voice/electronic ink annotation system, pen computing system without use handwriting recognition, instead using of gestures and direct manipulation. Notes that subjects found most useful: Synchronized voice/sketching annotation in recorded whiteboard system. [PerryS89a] Perry, Tkla S. and Voelcker, John "Of mice and menus; Designing the user-friendly interface", IEEE Spectrum, September 1989, pp Historical view (circa 1989) of graphical user interfaces: Sketchpad in 1962, lightpen in 1960's, Graphical curve-tracer with a potentiometer, mouse in 1964 by Douglas Engelbart at SRI, Trackball. Sketchpad split screen into two windows. No mention of handwriting or gesture recognition, or of Rand tablet. [PersonalWriter89] Personal Writer, Inc. "Personal Writer 10SL/15SL product literature", 1900 Avenue of the Stars, Suite 2870, Los Angeles, California (see also Anatex). Copyright 1987/1988/1989 Previously called Anatex (which see) Adaptive handwriting recognition product for MacIntosh, trainable, using a spelling dictionary [PersonalWriter90a] Personal Writer, Inc. "Personal Writer. Pen and paper has come of age", 1900 Avenue of the Stars, Suite 2870, Los Angeles, California (see also Anatex). Copyright 1987/1988/1989 Previously called Anatex (which see) Slide presentation on Personal Writer: handwriting recognition product PW10SL Version 1.0 on IBM PC, other for Apple MacIntosh. Refers to tablet as "absolute mouse" (equivalent). [PetersonS90] Peterson, Scott "Draft patent application for "The Guide"", for Simulated Input Device User Interface, Wang Laboratories, as drafted by patent counsel. Patent description for Wang/Guide SID simulated device user-interface, application independent [PfleegerCP89a] Pfleeger, Charles P. "Security in Computing", Prentice-Hall, 1989 Tutorial and reference on encryption and PKI, intrusion detection, user authentication, secure/trusted computer systems, copy protection, DRM, legal issues of (excessive) computer security [PhillipsM90] Phillips, Mark J; Philbin, Patrick T. and Blesser, Barry A. "Coordinated Determining Device using Spatial Filters", United States Patent 4,963,703, October 16, 1990 Electromagnetic digitizing tablet/pen with tilt measurement and stylus tilt (mis-named as parallax) correction: Cited later for prior art on acoustic/ultrasonics tablets [PlaisantC90a] Plaisant, Catherine and Wallace, Daniel "Touchscreen Toggle Switches: Push or Slide? Design Issues and Usability Study", Report HCIL-90-08, CS-TR-2557, CAR-TR-521, November 1990, U. Maryland Institute for Advanced Computer Studies Touchscreen virtual toggle and slider switches: users spontaneously associate slider switch with a control function. Compare to Apple slide-to-unlock. [PlamondonR89a] Plamondon, R. and Lorette, G. "Automatic signature verification and writer identification -- the state of the art", Pattern Recognition, Vol 22 No 2, March 1989, pp Review of all current commercial and research signature verification systems -- specificallyexcludes patents [PlamondonR90a] Plamondon, R. and Yergeau, P. "A System for the Analysis and Synthesis of Handwriting", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp [Poqet89] Poqet Computer Corporation "Poqet Computer Corporation press release", October 2, 1989 Rumored portable touchscreen product: no character recognition [Quam89] Quam, David L., Williams, George B., Agnew, Jeffery R., and Browne, Patricia C. "An Experimental Determination of Human Hand Accuracy with a DataGlove", Proceedings of Human Factors Society 33rd Annual Meeting, October 16-20, 1989, Denver Colorado, pp Pointing accuracy with human hand, but open loop (no coupling to a tablet digitizer/display tracking crosshair, for example). [QuerciaV89] Quercia, Valerie and Ro'Reilly, Tim "X Window System User's Guide for X11 R3 and R4 of the X Window System", Three volumes: O'Reilly & Associates, 1989 (hardcopy book) X-Windows documentation: mouse/pointer based windows interface, can be used with a tablet digitizer with a puck with at least three functional buttons, for mouse-button menus. No use with a stylus. Mousebutton menus can be re-defined. electronic copy also. [QuerciaV90a] Quercia, Valerie and Ro'Reilly, Tim "X Window System User's Guide OSF/Motif Edition", Three 8:26:33 PM]

187 ( ) History of Pen and Gesture Computing volumes: O'Reilly & Associates, 1990 (hardcopy book) X-Windows documentation for OSF/Motif windows manager system: mouse/pointer based windows interface, can be used with a tablet digitizer with a puck with at least three functional buttons, for mousebutton menus. No use with a stylus. Mouse-button menus can be re-defined. electronic copy also. [RecognitiveSciences89] Litvin, Y. and Kuklinski, T. "Recognitive Sciences Provides Cost Effective OCR Solutions for OEMs", company profile, Principals: Ted Kuklinski and Yuri Litvin. 323 Andover Street, Wilmington, Massachusetts, Kuklinski's and Yuri's O.C.R for handwritten and embossed characters [RegtienPPL89a] Regtien, P.P.L. "Silicon Tactile Image Sensors", in "Traditional and Non-Traditional Robotic Sensors", Proc. NATO Advanced Research Workshop on Traditional and Non-Traditional Robotic Sensors, Maratea, Italy, August 28-September 2, pp Review of 2D tactile/touch/touchpad sensor technologies: displacement-based / force include elastomeric deformable plastic sheets, piezoresistive, piezoelectric. [RevelleGL90a] Revelle, Glenda L. and Strommen, Erik F. "The effects of practice and input device used on young children's computer control", Collegiate Microcomputer, Vol 8 No 4, Nov. 1990, pp Joystick, mouse, trackball pointing devices: cites to earlier studies using touchscreen digitizer tablets. Child users always press and hold button for long time when intending to select, click on button unintentionally while thinking what to do next. [RobertG90a] Robert, Gregory; Chase, David; and Schaefer, Ronald "Software Licensing Management System", United States Patent 4,937,863, June 26, 1990 Software licensing: program checks in with licensing service, that keeps track of usage. If denied, check group license table. License released by program when done. [RobertsonGG89] Robertson, Geroge G., Card, Stuart K., and Mackinlay, Jock D. "The Cognitive Coprocessor Architecture for Interactive User Interfaces", Proc. 2nd Annual ACM SIGGRAPH symposium, Williamsburg Virginia, 1989, pp User-programmable buttons, RoomsButtons experimental system extended (with other objects) to support gestures. Visual feedback on gestures? Describes a 3D and animated user interface system involving rooms that uses interactive objects (buttons) which are responsive to multiple gestures (press, rubout, check, and throw). [RosenbergJ90a] Rosenberg, Jarret (moderator): Asente, Paul; Linton, Mark; Palay, Andrew "X Toolkits: the Lessons Learned", Proc. UIST '90, pp X Windows needs a toolkit. Constraint and error checking in toolkit means windows implementation can be simpler, since it finds bugs and reduces need for run-time checking or complications. Like a good compiler does for an instruction set. [RubineD90a] Rubine, Dean and McAvinney, Paul "Programmable Finger-Tracking Instrument Controllers", Computer Music Journal, Vol 14 No 1, Spring 1990, pp (partial copy) multi-touch musical instrument touchscreen interface: see also VideoHarp, SensorFrame 92. [RutledgeJD90a] Rutledge, Joseph D. and Selker, Ted "Force-to-Motion Functions for Pointing", Proc. INTERACT '90, IFIP, 1990, pp isometric (force-sensing) joystick in keyboard next to J key is faster than using mouse, because user need not move hand from keyboard. Velocity is non-linear with force: dead-zone near zero, etc. See also IBM keyboard joystick. [RyderJH90a] Ryder, John Hoyt Sr. and Smith, Susanna Ross "A method of verifying receipt and acceptance of electronically delivered data objects", European Patent Specification EP B1, June 9, 1990 DRM: electronic verification that user accepts software/use license, entering acceptance record into object/file/program. [SAC89a] Science Accessories Corporation "Model GP-7 Grafbar Mark II Sonic Digitizer", Science Accessories Corporation, 1989 Acoustic digitizer, two microphones in instrumentation bar at top of digitizing area up to 20x20 inches button cursor or stylus. 70 KHz acoustic signal generated in stylus/puck. Inking stylus, support circuit diagrams. [SabourinR89a] Sabourin, Robert and Plamondon, Re'jean "Segmentation of Handwritten Signature Images: A Structural Approach", unpublished manuscript, July 26, Laboratoire Scribens, Ecole de technologie Superieure, Departement de Production Automatisee, Montreal, Quebec, Canada. Reviewed for PAMI and Henry Baird: extraction of signatures from scanned documents prior to verification [SabourinR90a] Sabourin, Robert and Plamondon, Rejean "Progress in the Field of Automatic Handwritten Signature Verification Systems Using Gray-Level Images", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp 1-12 Signature verification with OCR: can tell photocopies from originals 8:26:33 PM]

188 ( ) History of Pen and Gesture Computing [SandigeRS90a] Sandiga, Richard S. "Modern Digital Design", McGraw-Hill, 1990 (hardcopy book) Upper-level textbook on digital design. Focuses on standard commercial components (e.g. no description of multiply) [SantoriM90a] Santori, Michael "An instrument that isn't really: LabVIEW's software approach to instrumentation systems yields an intuitive programming tool for test needs", IEEE Spectrum, August 1990, pp LabView graphical programming language for software simulation and control of virtual laboratory instruments, describing dataflow programming. Introduction gives history of LabVIEW to Quoted in JohnsonGW94a. [SatoK89] Sato, Koji, Kurakake, Masahiro, and Fujiwaki, Kazuo "Optical Character Reader", United States Patent 4,797,940, January 10, 1989,assigned to Sumitomo Electric Industries, Osaka, Japan (abstract only). Hand-held OCR scanner which can read one line at a time, with hardware to segment into separate characters [SatsutaM90a] Satsuta, Mayasuki and Nakamura, Shoji "Multi-Window Display Control System", Japanese laidopen patent application JP , June 14, 1990 Semi-transparent overlapping windows using stippling / half-tone using XOR to flip pixel bits: offset if multiple windows so can see each. Double-click gesture to bring lower window to top. [ScanOptics90] Scan-Optics, Inc. "Easyreader 1720 OCR product", press release, February 1, 1990 OCR scanner product: programming via format control sheets, filled out by hand to define fields, field lengths, numeric range context. [Scenario89a] Scenario, Inc. and Visible Systems Corp. "Easy Touch/Touch Translator", structured UI design tool, PC Week, Dec 4, 1989, vol 6 no 48, p 73 Touch-screen user-interface design tool: Pen-based UI SDK. [Scenario89b] Scenario Inc "Dynabook product announcement file", October 1989, Boston, Massachusetts. Press kit for Touch-screen portable (sort of) CD-ROM product: no handwriting recognition yet. File contains additional references on Dynawriter (change of product name) with handwriting recognition. detachable resistive-film (SAW) touchscreen -- most likely EloGraphics? Name taken from Alan Kay's Dynabook? Scenario, Inc. Technical Products Summary: include consulting, Touch API/SDK utilities, Touchscreen Designer GUI developer, ELODEV TSR driver supports keyboard buffer stuffing, access to speaker, pressure data (??), user-specific calibration (left-handed writers?). Touch Translator / Dynavision for converting touchscreen menus to keyboard commands -- compare to PenCept PenPad utilities. Graphics Touch Zone Generator TouchUp, TouchBack [SchlosserPA89a] Schlosser, Philip A. and Kable, Robert G. "Electrographic apparatus", United States Patent 4,853,493, August 1, 1989 Single-resistive sheet capacitive digitizer: using resistors at contact points at edges to correct mostly for pincushion/reverse pen-cushion distortion in position. Single on one side, ground on other. [SchneidawindJ90a] Schneidawind, John "Software pioneers are reunited at hot new Arizona company", Press report, 1990 (publication not known) Bob Frankston and Dan Bricklin join Slate Corporation: pen computing applications (PenPoint, W4PC). Investment by Mitch Kapor [Scriptel90a] Scriptel "Press Information on PenWriter", Press reports circa 1990 Press reports on Scriptel: includes Ohio Gov. Richard Celeste using Scriptel transparent tablet as telautograph to sign bill concerning electronic signatures. [SearsA90a] Sear, Andrew; Plaisant, Catherine; Shneiderman, Ben "A new era for touchscreen application: High precision, dragging icons, and refined feedback", U. Maryland HCI Lab., Dep.t of Comp.Sci, CAR-TR-506 / CS-TR- 2487, June Also in Advances in Human-Computer Interaction, Hartson and Hix, 1993 Review of touchscreen user interface considerations, including hardware trade-offs. Mentions first-touch versus lift-off for selection and pointing. Touchscreen keyboards somewhat slower typing than regular keyboards, but fast than people thought. Accuracy of rotation gestures improved by longer lever distance by user. Microsoft UnMouse. See: multi-touch "has been explored", looking for applications when it becomes commercially available: touchscreen manufacturers will be important. SAW may be able to detect as least two touches. On-screen touchscreen keyboards more flexible e.g. Dvorak or other keyboard layouts (see also Kaplow). Directness, no moving parts, no extra desk or keyboard space, speed, ease of use benefits of touchscreens. Refers to multi-touch tablet research, but in context of preventing inadvertent second touches. [SeymourJ90a] Seymour, Jim "The "Slate PC" Wave Is About To Break", PC Magazine, October 16, 1990, page Editorial that pen-computing PCs (like Palm?), as small portable devices making only minor use of handwriting character recognition, would be successful. Mentions GRiD, Go/PenPoint, Momenta. No mention of Microsoft Pen-Windows. [ShearVH90a] Shear, Victor H. "Database Usage Metering and Protection System and Method", United States 8:26:33 PM]

189 ( ) History of Pen and Gesture Computing Patent 4,977,594, December 11, 1990 Metered database usage: database is encrypted, user's charged for amount of data fetched. Database "self destructs" if tampered with: delete decrypting key [Shu89] Shu, Joseph Shou-Pyng "One-pixel-wide Edge Detection", Pattern Recognition, Vol 22 No 6, 1989, pp Edge detection/skeletonization/line-thinning using a 3x3 pixel filter, similar to Sobel edge detector filters, Robert 2x2 edge detectors, combined with heuristic (minimum-cost) search Edge-streak encoding, similar to chain codes for edges [Signaware89] Signaware Corp. "Software Description", 1989, 300 South Duncan Avenue, Suite 273, Clearwater, Florida 34615, Internal Wang memo on signature verification vendors: one in Florida, one in Ireland [SiliconGraphics90a] Silicon Graphics "Silicon Graphics Workstation - Desktop", ACM CHI 1990 Issue 56, Video - Silicon Graphics Workstation. Available at open-video.org, 2010 Shows highlighting on selection and motion during drag in GUI for Silicon Graphics Workstation Unix, for file explorer window. Drag and drop, drop-available highlighting when dragging over a target. [SimonJC90a] Simon, J.C. and Baret, O. "Handwriting Recognition as an Application of Regularities and Singularities in Line Pictures", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp [SinghG89a] Singh, Gurminder and Green, Mark "Chisel: A System for Creating Highly Interactive Screen Layouts", Proc. UIST '89, 2nd Annual ACM SIGGRAPH symposium on User Interface Software and Technology, pp Automatic screen layout in UIMS user interface definition: color and window resources management [SinghG89b] Singh, Gurminder "Automating the Lexical and Syntactic Design of Graphical User Interfaces", Ph.D. Thesis, U. of Alberta, Canada, 1989 UIMS development system. UI defined declaratively using a graphic editor / layout editor not for the user interface appearance, but for the interaction structure and positional constraints of the UI. [SinghG90a] Singh, Gurminder; Kok, Chun Hong; and Ngan, Teng Ye "Druid: A System For Demonstrational Rapid User Interface Development", Proc. UIST '90, 3rd ann. ACM SIGGRAPH symposium on User Interface Software and Technology, New York, 1990, pp UIMS development system for prototyping. User defines UI elements graphically (direct manipulation), animates them manually, system "learns" the UI behavior to simulate. Prototype implemented using Motif GUI. [Sklarew89a] Sklarew, Ralph "Handwritten Keyboardless Entry Computer System", European Patent Application A2 Abstract refers to transparent sensing surface over display, electronic ink, character recognition, and learning mode Title and abstract read almost identically to other Sklarew patents [Sklarew90a] Sklarew, Ralph "Handwritten Keyboardless Entry Computer system", United States Patent 4,972,496, November 20, 1990 U.S. Patent application on handwriting recognition user-interface, arm-held "slate" type pen computer. Patent description of prior art for Linus product, Grid patent on user interfaces: transparent digitizer over display for handwriting input, with character recognition Abstract refers to transparent sensing surface over display, electronic ink, character recognition, and learning mode/ trainable recognition. Cited for Unistroke character set. Transparent front-mounted digitizer tablet/touchscreen: cites advantages (stylus tilt, etc.) over rear-mounted tablets. Refers to trainable recognition for handwritten characters and symbols, but not to macros or gestures. [SklarewR90b] Sklarew, Ralph "Correspondence file: Ralph Sklarew", Personal correspondence, Ralph Sklarew, 1990 Technical/hardware descriptions of keyboardless touchscreen portable computer with virtual on-screen keyboard [SmallCH90a] Small, Charles H. "Inexpensive software reads printed material", Electronic Design News, January 18, 1990, pp Review of OCR scanner companies: Caere Corporation, Inovatic, Olduvai Corporation, Xerox Imaging Systems. [Sony90] Sony Corporation "Sony hand-held computer press release", March, 1990 Hand-held slate computer product from Sony Corporation. [SpitzG89a] Spitz, Gabriel "The Preferred Pen Activation Force for the Low Resolution Freestyle Tablet", Research report 14 July, 1989, Wang Laboratories, 1 Industrial Avenue, Lowell, Massachusetts Internal Wang report on stylus pen force/pressure for handwriting force [SpitzG90a] Spitz, Gabriel and Akagi, Kenichi "Stylus Button Location", research report 3 January, 1990, Wang Laboratories, 1 Industrial Avenue, Lowell, Massachusetts :26:33 PM]

190 ( ) History of Pen and Gesture Computing Internal Wang report on barrel-switch/side-button on digitizer tablet stylus [SrihariSN90a] Srihari, S.N. "Reading Unconstrained Handwriting with Bounded Context", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp Overview of some recognition context problems: project was OCR for handwritten Zip codes, address blocks [Stentiford89] Stentiford, Frederick W.M. "Pattern Recognition System", United States Patent 4,817,171, March 28, 1989, assigned to British Telecommunications public limited company, Great Britain. OCR [StringaL90a] Stringa, L. "A Structural Approach to Automatic Primitive Extraction in Hand-printed Character Recognition", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp (abstract only) [SuenCY90a] Suen, C.Y., Nadal, C., Mai, T.A., Legault, R. and Lam, L. "Recognition of Totally Unconstrained Handwritten Numerals Based on the Concept of Multiple Experts", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp [Summagraphics89a] Summagraphics Corporation "Summagraphics The Summasketch II ADB / The Microgrid II Series MacIntosh SE and II Version", Summagraphics, 1988, 1989 Electromagnetic digitizers with stylus or puck: 16 button cursor/puck, Laser Interferometer System (for production testing?), Compatible (mouse replacement) with Apple computers, menu picking. Full Multiple Monitor Support tablet/digitizer can be mapped to any contiguous group of monitors (on Apple Desktop Bus). Independent (of application) control panel configuration of scaling, offset, button mapping, and tablet orientation (90-degrees?). File includes data sheet on SummaSketchII for IBM PC. [Summagraphics89b] Summagraphics Corporation "Decisions: Summagraphics goes Hollywood", Summagraphics 1989 Summagraphics tablets in movies/films as technology prop. Mentions Dr. Halo III graphics application software. [Summagraphics89c] Summagraphics Corporation "Draw on our experience -- Summagraphics tablet modes", Summagraphics 1989 Summagraphics tablet product information: SummaSketch, Bit Pad Plus for IBM/MacIntosh: Windows driver (mouse), Autodesck Device Interface (driver). Optical mouse, Microgrid II drafting-table size. Most are electromagnetic, but one CR Charge Ratio capacitive tablet. Stylus and cursor (puck). [Summagraphics90a] Summagraphics Corporation "Summagraphics MM II 1201 Graphics Tablet Technical Reference", Summagraphics January 1990 Summagraphics electromangetic tablet: modes Point Mode, Remote Request Mode (polled input only), Stream Mode, Switch Streem Mode, Report Rate, Increment Mode, Axis Update Mode. Device driver guidelines. Stylus, 4-button and 16-button cursor (puck). Compare increment mode with press-and-hold distance. Manual setting of origin, [SunMicrosystems90a] Sun Microsystems "Open Windows Developer's Guide 1.1, User's manual", Sun Microsystems, 1990 X-Windows-like windows management system with graphical objects. pop-up menus? drag gesture? Popup virtual keyboard? Cited in Pier92 [SuzukiT90a] Suzuki, T. and Mori, S. "A Thinning Method Based on Cell Structure", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp Line-thinning / skeletonization [Taguchi89] Taguchi, Yoshinori; Yamanami, Tsuguya "Coordinate Input Device with Display", United States Patent 4,845,478, July 4, 1989 Patent on Wacom electromagnetic / magnetostrictive digitizer using cordless stylus. Does not describe sensor or switch for detecting stylus down or contact with surface. [TakahashiH90a] Takahashi, Hiroshi "Information Processing Apparatus", United States Patent 4,954,967, September 4, 1990 Scrolling bar in hardware using a linear array of microswitches to detect where finger is. Microswitches implemented as X and Y conductors, a hold in a cover sheet lets finger press the layers together at a defined location. [TangJC90a] Tang, John C. and Minneman, Scott L. "Videodraw: A Video Interface for Collaborative Drawning", Proc. CHI '90, April 1990, pp Video whiteboard system: actual video image of other user (or of user's hands) shown ion display aligned with whiteboard touchscreen, so that writing surface and video not separate. Actual drawing ink used: first 8:26:33 PM]

191 ( ) History of Pen and Gesture Computing user cannot use second user's ink, but this was advantageous for users to draw separate layers of drawing. [TaniguchiT90a] Taniguchi, Takashi; Kurasaki, Shoichi; and Oka, Koichiro "Pressure-sensitive Tablet", United States Patent 4,963,417, October 16, 1990 Pressure/force-sensitive tablet, resistive/conductive sheets with pressure-sensitive elastomeric layer between: small conductive particles in compressive material. Transparent on polyethersufones and polyesters, polyacrylic or polycarbonates for rigid member. ITO, tin-oxide, gold, paladium, chromium for electro-conductive membrane. 76% transmissivity perceived as transparent overlay. Claim 1 gives mathematical formula for deformabilility. [TaniishiS89a] Taniishi, Shinnosuke "Acoustic coordinate input device using a roughened surface to attenuate the surface wave component", United States Patent 4,853,496, August 1, 1989 Acoustic digitizer with roughened surface to reduce the surface acoustic wave (SAW) component: detection uses internal waves in the glass. Acoustic signal is generated by vibrating stylus. [TappertCC89] Tappert, Charles C. and Jeanty, H.H. "A Study of Several Accuracy-Improvement Methods for a Handwriting Recognition System", IBM Research Report, IBM T.J. Watson Research Center, P.O. Box 704, Yorktown Heights, NY 10598, November 1989 Most promising feature algorithm for improving recognition over elastic matching is pairwise comparison (but NOT tuned by hand!) Features (beyond elastic matching) are heights of endpoints, relative stroke positions (min and max extents), line extension, relative width, etc. Most handwriting recognition error for other systems was transducer/stylus % error, user error (user wrote "wrong")! Cites on study that handwriting on integrated tablet/display (electronic ink) sloppier than on separate tablet, much sloppier than on paper [TappertCC90a] Tappert, C.C., Suen, C.Y., and Wakahara, T. "On-line Handwriting Recognition - a Survey", manuscript in preparation, to appear in IEEE Trans. Pattern Analysis and Machine Intelligence: C.C. Tappert, IBM T.J. Watson Research Center, Yorktown Heights, N.Y Historical development of elastic matching for script, discrete handwriting character recognition. See also 1988 conference paper. Commercial handwriting input systems listed. [TappertCC90b] Tappert, Charles C. "Rationale for Adaptive Online Handwriting Recognition", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp [TappertCC90c] Tapper, Charles C., Suen, Ching Y,. and Wakahara, Toru "The State of the Art in On-line Handwriting Recognition", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol 12 No 8, August 1990, pp 787-ff Comprehensive review of handwriting/gesture recognition technologies as of Gives historical remarks pen-computing in the 1950s, 1960's, 1970's, 1980's. Cited in Agulnick 295 patent. [TaxtT89a] Taxt, Torfinn; Flynn, Patrick J.; and Jain, Anil K. "Segmentation of Document Images", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol 11 No 12, December 1989, pp Statistical classifier to separate text and background images on documents Document segmentation using contextual Bayes, Besag's, relaxation, Owen and Switzer, versus Haslett's classifiers: which is best varies [TaxtT90a] Taxt, Torfinn; Olafsdottir, Jorunn B.; and Daehlen, Morten "Recognition of Handwritten Symbols", accepted for Journal of Pattern Recognition, 1990 O.C.R. on handwriting with 98.6% and 97.7% accuracy O.C.R. of handwritten map symbols, using several edge detection, spline curve fits, Fourier expansion, and statistical classification (lower-case alphanumerics) O.C.R. of handwritten lower-case alphanumeric using circular graphs [Teh89] Teh, Cho-Huak and Chin, Roland T. "On the Detection of Dominant Points on Digital Curves", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol 11 No 8, August 1989, pp Scale-space filtering (size-dependent) of small features in machine vision. Keywords: chain code, curve tracing, polygonal approximation, tangential deflection. Points of high curvature are dominant features for pattern recognition. Lots of references on chain codes and curve tracing for machine vision. [Termiflex89] Termiflex Corporation "TouchGraf Keyless Keypad: the Next Generation of Flexible Terminals", 316 Daniel Webster Highway, Merrimack, NH 03054, March 1989 Handwriting capture terminal product, separate digitizer built into a hand-held terminal: digitizer/touchpad and display separate [TeulingsHL90a] Teulings, H.-L. "Handwriting movement control - Research into different levels of the motor system", Doctoral Thesis, Nigmeegs Instituut voor Cognitive-onderzoek en Informatietechnologie, Nijmegen, 1990 [TeulingsHL90b] Teulings, H.-L. and Schomaker, L.R.B. "A Handwriting Recognition System Based on the Properties and Architectures of the Human Motor System", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp 195-ff. On-line recognition with a tablet give better results than off-line OCR grayscale recognition, because motor 8:26:33 PM]

192 ( ) History of Pen and Gesture Computing information available. [TiburtiusB89a] Tiburtius, Bernd "Transparente Folientastaturen", Feinwerktechnik und Messtechnik, July 1998, no 7, pp Overview of transparent touchpanels/touchpads. Capacitive and resistive sheet ("analog"), capacitive and resistive sensor array ("discrete"). Flexible materials (for Elographics-style resistive sheet), conductive silver ink (Silberleitfarbe) on polyester or polycarbonate substrate. [TogawaF90a] Togawa, Fumio and Hirose, Hitoshi "Handwritten Character-Recognizing Apparatus for Automatically Generating and Displaying Character Frames", United States Patent 4,953,225, August 28, 1990 boxed/boxless recognition with boxes generated to right of current/previous handwriting character as user writes: automatic spacing of characters, stroke parsing / segmentation. [Toshiba89] Toshiba America Information Systems, Inc. "Toshiba PenPC product information", Toshiba America press release November 3, 1989, Pre-announced portable handwriting recognition product for ComDex 1989: kludged together from Linus, other parts? (Includes separate announcement about Linus Technologies (Sklarew). DOS 3.3 OS. Mentions mouse emulation with touchscreen digitizer. Does not mention PenPoint or Pen Windows tablet computers. [TsuchiyaH90a] Tsuchiya, Huraki "Electronic Book", European Patent EP A2, October 3, 1990 Electronic book with update/loading by wireless communications or floppy-disk/laser card, etc. Described as a personal computer deprived of functionality such as full keyboard and printing. Page-turning user interface is small set of dedicated keys/buttons. Annotation? Shows fold-out device with two-part display; compare with Refalo? [Uncapher89] Uncapher, Kieth "Interview with Kieth Uncapher", Arthur L. Norberg, 10 July, Charles Babbage Institute, Center for the History of Information Processing, University of Minnesota, Minneapolis. Available at Interview with Kieth Uncapher of RAND Corporation: mentions GRAIL system with electronic ink on the Rand tablet. Describes use of tablet for flow-chart and programming text entry. Describes invention of the disc drive (to replace the drum drive) as a result of this project. Mentions a video film showing the Rand tablet and handwriting recognition in operation in the 1960's. Rand tablet invented by Tom Ellis around 1966: also an earlier tablet by Herb Teager at Boston College (Graphacon?). Other work by Gabe Groner, and Bill Sibley. [Unser89] Unser, Michael and Eden, Murray "Multiresolution Feature Extraction and Selection for Texture Segmentation", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol 11 No 7, July 1989, pp Segmentation of textured images for machine vision: refers to Fisher's multiple linear discriminants and Karhunen-Loe`ve transform [VanLeeuwen89] Van Leeuwen, H.C. "A development tool for linguistic rules", Computer Speech and Language, Vol 3 No 1, January 1989, pp Phonology, grapheme-to-phoneme conversion rules for text-to-speech [Vosburgh89] Vosburgh, A.M. and Brown, N.R. "Following the bouncing ball: An evaluation of a self-pacing enrollment interface for a large-vocabulary speech recognition system", Research Report RC-64372, IBM Thomas J. Watson Research Center, Yorktown Heights, New York, 1989 Brown,NR89 on user interface / enrollment for adaptive recognition [Wacom89a] Wacom, Inc. "Wacom PC Installation and Utilities Manual", Park 80 West Plaza II, Saddle Brook, NJ 07662, November 1989 Describes emulation and multiple command sets for digitizer controller/driver Digitizer tablet with absolute and relative (i.e. mouse) modes [Wacom89b] Wacom, Inc. "Super Digitizer product information", West 115 Century Road, Paramus NJ 07562, August 1989 Product literature on Wacom force/pressure-sensitive pen stylus and cordless pen stylus tablet. Product literature describing Wacom cordless pen stylus using resonating coil in tablet digitizer CIC memo lists a katakana and Roman and kanji system under development. Digitizer tablet with absolute and relative (i.e. mouse) modes. [Wacom89c] Wacom, Inc. "Wacom SD x12 Cordless Digitizer / Super Digitizer, SD x15, SD x25", Wacom, 1989 Digitizer tablet with cordless, battery-free stylus and or cursor (puck). SP-300 pressure-sensitive stylus. SD-421L has electrostatic adhesion surface to hold papers. Proximity == cursor reading height. Correspondence on WT-460M digitizer for handwriting character recognition -- other clients? [Wacom90a] Wacom, Inc. "Wacom Sapphire tablet and mouse", Wacom, 1990 (date approximate) (physical device) Electromagnetic tablet (Wacom), small format, pressure/force-sensing cordless stylus, also includes "mouse" -- mouse-styled wireless puck. [WakimotoK90a] Wakimoto, K., Shima, M., Tanaka, S., and Maeda, A. "An Intelligent User Interface to an Image 8:26:33 PM]

193 ( ) History of Pen and Gesture Computing Database Using a Figure Interpretation Method", 10th International Conference on Pattern Recognition, Atlantic City, New Jersey, June 16-21, 1990 [WalkerJT90a] Walter, Jame T. and Kausch, Marvin L. "Video Overlay Image Processing Apparatus", United States Patent 4,954,970, September 4, 1990 Overlay image on screen, using separate table *degree* of mixing (0 transparent to 1 opaque) for each pixel.: 1 is complete masking or protection. [WangJS89] Wang, J.S. "Electronic text-graphic editing system", United States Patent 4,829,470, May 9, 1989, assigned to IBM Corporation. Text-graphic editing system includes mixed object editor which flows text around into irregularly shaped objects on page (for a mark-up language, like runoff for text composition in documents) [WangLaboratories89a] Wang Laboratories "The Wang Freestyle System", Wang Laboratories product literature Wang Freestyle: pen-computing electronic-ink and voice annotation, no character recognition. Used customized Hitachi tablet, with first (?) double-ended stylus with an eraser on the back end. Pure electronic ink and selection application system, with synchronized voice: interesting case of human communication. Standalone Demo diskette (5.25"). [WangLaboratories89b] Wang Laboratories "Freestyle: A Multimedia Conferencing System", Video: ACM CHI 1989 Issue 45 Video Presentations: Available (2009) from videoid=8067&surrogate=clip Wang Freestyle: pen-computing electronic-ink and voice annotation, no character recognition. [WangLaboratories90a] Wang Laboratories "Freestyle: Wang Freestyle/Light User's Guide, Release 1.21", Wang Laboratories, Lowell Massachusetts Wang Freestyle: pen electronic-ink and voice annotation, no character recognition. Used customized Hitachi tablet, with first (?) double-ended stylus with an eraser on the back end. Describes image-only document (voice/electronic ink/imaging) data base using no handwriting recognition. No use of gestures. Cited by Bill Buxton. [WangLaboratories90b] Wang Laboratories "Freestyle: Take a look at the personal communication system you already know", Wang Laboratories, Lowell Massachusetts 5.25" floppy demo of Wang Freestyle, with brochure explaining the Freestyle UI: pen electronic-ink and voice annotation, no character recognition. [WangLaboratories90c] Wang Laboratories "Equipment/Device: Wang Freestyle miniature keyboard ca. 1990", Wang Laboratories (physical device) Small physical keyboard with pressure switches, suitable for activation with a stylus. Full 102-key keyboard, including function and shift keys. Intended to be used in conjunction with Freestyle tablet digitizer for Freestyle UI. Likely a prototype. [WangLaboratories90d] Wang Laboratories "Wang Freestyle Installation Guide, Release 1.2", Wang Laboratories 1990 (Partial copy) Installation guide for PC version of Wang Freestyle, handwriting/direct-manipulation office software with whiteboard and audio recording. [WangPSP89a] Wang, P.S.P. and Zhang, Y.Y. "A Fast and Flexible Thinning Algorithm", IEEE Trans. Computers, Vol 38 No 5, May 1989, pp Compares and shows results of different skeletonization/thinning algorithms for character and arbitrary shapes [WangPSP89b] Wang, P.S.P. "A New Approach for Chinese Handwriting Recognition", Proc. Int. Conf. on Electronic Imaging, Pasadena, pp , 1989 On-line (D.C.R.) recognition with very little system training/learning for Chinese. Chinese recognition using stroke angles in 22.5 degree ranges (chain codes?) [WardJR89a] Ward, Jean Renard "Testing tablet characteristics", Internal Report, Wang Laboratories, May 1989 Technical instructions on testing tablets to reveal bad behaviors for electronic ink, handwriting character recognition Describes differential vs. absolute error, spirograph tests, etc. [WardJR89b] Ward, Jean Renard "Tablet error mechanisms and performance analysis for Freestyle", Internal Report, Wang Laboratories, May 1989 Technical instructions on testing tablets to reveal bad behaviors for electronic ink, handwriting character recognition Describes differential vs. absolute error, spirograph tests, etc. [WardJR90a] Ward, Jean Renard and Kabel, Robert "The "How to" of Electronic Ink: Tablet Error Mechanisms and Performance Analysis for Handwriting Capture", Research report, Wang Laboratories, July 1990 Descriptions of "bad behaviors" in digitizing tablets, engineering review of the effects, test procedures for showing the bad behaviors [WardJR90b] Ward, Jean Renard "One View of On-going Problems in Handwriting Character Recognition", in Frontiers in Handwriting Recognition, C.Y. Suen, editor, Proceedings of the Conference on Frontiers in Handwriting Recognition, Concordia University, Montreal Canada, April 2-3, 1990, pp :26:33 PM]

194 ( ) History of Pen and Gesture Computing Biggest obstacle to on-line handwriting recognition/gesture technology is lack of developed applications. [WareC89a] Ware, Colin and Baxter, Curtis "BAT Brushes: On the Uses of Six Position and Orientation Parameters in a Paint Program", Proceedings of 1989 CHI conference, 1989, pp User-interface on simulating a brush with a stylus using stylus rotation angle, x/y/z digitizer tablet [WatsonJS89a] Watson, James S. "Ground Switching Technique for Silkscreened Digitizer Grids", United States Patent 4,853,499, August 1, 1989 Ground switching in transparent electromagnetic digitizer using ink-on-glass, "passive stylus". Two-layer grid. [Weideman89] Weideman, W.E., Manry, M.T. and Yau, H.C. "A comparison of nearest neighbor classifier and a neural network for numeric handprint character recognition", IEEE International Conference on Neural Networks, Washington, D.S., 1989 Martin,GL90 [WenzelEM90a] Wenzel, Elizabeth M.; Fisher, Scott S.; Stone, Philip K; and Foster, Scott H. "Virtual Acoustics Displays", Proc. of Visualization '90, IEEE, San Francisco, October 23-26, 1990, pp D audible/acoustic display coordinated with real-time gesture, three-dimensional visual objects. Auditory cues/icons using Auditory Cue Editor. Artificial/virtual reality in general user interface. Three-dimensional psychophysical/psychoacoustic audio synthesized. Numerous references to Buxton, Foley, Egan for audio cues. [WhiteSR90a] White, Steve R. and Comerford, Liam "ABYSS: An Architecture for Software Protection", IEEE Trans. on Software Engineering, Vol 16 No 6, June 1990, pp Software copy protection / rights authorization based on passing rights-to-execute vi a token. Enforcement by protected/secure (logically, physically, procedurally secure) processors, encrypted/secure communications between secure processors. Rights-to-execute can be time-limited, dependent on callers, etc. Use count enforced by limited number of use-once rights-to-execute. Can be extended to enforce additional terms of use (contract terms). [WiechaC89a] Weicha, Charles (chair); Bois, Sstephen; Green, Mark; Hudson, Scott; and Myers, Brad "Panel: Direct manipulation or programming: how should we design interfaces?", Proc. 2nd ACM SIGGRAPH Symp. on User Interface Software and Technology, UIST '89, pp Direct manipulation to create user interface vs. text specification: real question is how to combine these two. [WolfCG89a] Wolf, Catherine G., Rhyne, J.R., and Ellozy, H.A. "The paper-like interface", in "Designing and Using Human-Computer Interfaces and Knowledge-Based Systems", G. Salvendy and M.J. Smith, Eds., Elsevier Science Publishers, Amsterdam, 1989, pp Description of paper-like interface PLI projects at IBM: electrostatic/capacitive digitizer (Scriptel) over LCD display, exploration of gesture input. Video demonstration shown in IBM lobby. Gestures described as handmarkings for user-interface in spreadsheet (Lotus) and drawing applications. Music editing demonstration modelled after Buxton. Two-dimensional mathematics editing. Note: video demonstration filed under Ellozy88a. [WolfCL90] Wolf, Chris L; Maikowski, Michael F.; Martin, Winston N.; and Lindsley, Walter E "System for selectively modifying code generated by a touch type keyboard upon detecting of predetermined sequence of make codes and break codes", United States Patent 4,937,778, June 26, 1990 Make use of roll-over on keyboard to use space bar as a shift key, by sequential decoding of make/break codes on keyboard keys [WolffenbuttelRF89a] Wolffenbuttel, R.F. and Regtien, P.P.L. "Integrated Tactile Imager with an Intrinsic Contour Detection Option", Sensors and Actuators, Vol 16, 1989, pp Capacitive grid touchpad/tactile imager, 9x9 pressure-sensitive capacitors with compression of nonconductive natural rubber layer between top square and common conductive rubber electrode. [WooA90a] Woo, Andrew; Poulin, Pierre; and Fournier, Alain "A Survey of Shadow Algorithms", IEEE CGA November 1990, pp Rendering techniques for photorealistic shadows in virtual images. Some attempt to account for specular reflection, reflective properties of bumpy surfaces. Bump-mapping, particle shadows, polygonization [YamadaK89a] Yamada, K., Kami, H., Tsukumo, J., and Temma, T. "Handwritten Numeral Recognition by Multilayered Neural Network with Improved Learning Algorithm", Proceedings of International Joint Conference on Neural Networks, Washington, D.C., June 18-22, 1989, Vol II pp OCR for handwritten numerals using neural network. Gives samples of low-quality, but real-world handwriting test data for numerics for OCR. [Yamaguchi89] Yamaguchi, Mikio and Sato, Koji "Optical Character Reader", United States Patent 4,817,185, March 28, 1989, assigned to Samitomo Electric Industries, Limited, Osaka, Japan. Hand-held scanner for reading price tags [YoungDA90a] Young, Douglas A. "The X Window System: Programming and Applications with Xt: OSF/Motif 8:26:33 PM]

195 ( ) History of Pen and Gesture Computing Edition", Prentice-Hall, 1990 (hardcopy book) X-Windows programming: describes setting resource values (providing values) to affect the way a widget appears or behaves by specifying values for resources: XtSetValue, XtGetValues [YuSS90a] Yu, Shiaw-Shian and Tsai, Wen-Hsiang "A new thinning algorithm for gray-scale images by the relaxation technique", Pattern Recognition, Vol 23 No 10, October 1990 pp Uses chain codes context to preserve line straightness in line thinning: for sketch recognition? Straightness of lines very important in Chinese/Kanji character recognition [ZandenBV90a] Zanden, Brad Vander and Myers, Brad A. "Automatic, Look-and-Feel Independent Dialog Creation for Graphical User Interfaces", Proc. CHI '90, April 1990, pp Garnet system: Jade UI definition UIMS: UIMS supporting arbitrary looks and feels. Dialog box ore menu look and feel changed on the fly by switching look and feel databases. Jade obtains graphic information from (multiple) style files, example changing Garnet style dialog to OpenLook style, style can include changes to layout or graphics or adding elements, position of objects in dialog. Shows dynamicallychangeable themes? [ZhuangX89a] Zhuang, X., Haralick, R.M., and Joo, H. "A Simplex-Like Algorithm for the Relaxation Labeling Process", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol 11 No 12, December, 1989, pp biblio88 (Prev page) Main Page (Next page) biblio93 You must build up trust in a relationship, before you can start abusing that trust.... and that takes time. "Auf den Inhalt kommt es an." -- Walter Ulbricht 8:26:33 PM]

196 ( ) History of Pen and Gesture Computing History of Pen and Gesture Computing: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers References from the approximate years 1991 to This compilation and all annotations are copyright Jean Renard Ward, 1993, 2004, 2011, 2013, Permission is granted to use this information in publication, including confidential reports, provided that accompanying text clearly makes reference to the URL for this page, along with the statement: Source: Annotated Bibliography in On-line Character Recognition, Pen Computing, Gesture User Interfaces and Tablet and Touch Computers Copyright Jean Renard Ward Most of these are in my personal collection, either as physical items or electronic files. I can be contacted at Copyright Sun May 15 19:16:38 EDT Mirrors of this page can be found at: (Generally most up-to-date) biblio90 (Prev page) Main Page (Next page) biblio95 [ATTHobbit93a] A.T. & T. "ATT Hobbit pen computer running PenPoint (functional unit)", Personal collection of Jean Renard Ward, (May not be functional) PenPoint "Hobbit" pen computer from AT&T, includes manuals. Appears to be functional as of April 2011, PenPoint Operating System. [Abandonware93a] winworldpc.com "Microsoft Windows for Pen Computing 1.0.7z", wdl2.winworldpc.com, fetched 2012 Archive of installation file for Windows for Pen Computing 1.0 / Pen Windows -- self-install zip file is in electronic video files. [Aha92a] Aha! Software Corporation "aha! InkWriter with SmartInk reference card / Basics", Aha! Software corporation, 1992 Manual for InkWriter for PenPoint: electronic-ink scribble editing software, later acquired by Microsoft. Supports delays/deferred handwriting recognition, indexes ink by the recognition results. Editing gestures generally similar to PenPoint: X to delete, caret gesture to insert, B to make bold, press-and-hold to start selection. Compare with van Raamsdonk. [Aha93a] Aha! Software Corporation "aha! InkWriter Handbook", Aha! Software corporation, 1993 Manual for InkWriter for PenPoint: electronic-ink scribble editing software, later acquired by Microsoft. Supports delays/deferred handwriting recognition, indexes ink by the recognition results. Compare with van Raamsdonk. [Aha93b] Aha! Software Corporation "aha! InkWriter Summary", Aha! Software corporation, 1993 Summary of commands (not gestures_ for InkWriter for PenPoint: electronic-ink scribble editing software, later acquired by Microsoft. Supports delays/deferred handwriting recognition, indexes ink by the recognition results. Compare with van Raamsdonk. [AkeleyK93a] Akeley, Kurt "RealityEngine Graphics", Proc. SIGGRAPH 93, Computer Graphics vol 27, pp Subpixel image blending as alternative to simple alpha blending. Discussion of hardware acceleration of different commercial systems. [AllenD92a] Allen, Dennis "Pen-Input Systems", BYTE Magazine, April 1992, P :26:37 PM]

197 ( ) History of Pen and Gesture Computing Part of a series on pen-input systems and PDAs [AllenG93] Allen, Gregory "Data Input Grid for Computer", United States Patent 5,214,428, May 25, 1993 mechanical grooves restrict the motion of a digitizer stylus to octant chain-code directions: shows a set of single-stroke characters for recognition [AlsonS91a] Alson, Stewart "NCR takes Expo by storm with pen-based computer", InfoWorld, July 1, 1991 NCR 3125 pen computer runs either PenPoint for Windows for Pen-based Computing (sic). [Amstrad93] Amstrad93 "Amstrad PenPad PDA 600", Reference available at Z-80 handheld pen-computer/touchscreen, 1993, styled as fold-out notebook (Refalo?). Contains quotes from project manager Cliff Lawson, Mark Stevenson, Wulf Broening: software by German company Wiedmann Additional materials in file -- shows touch buttons on outside edge of display area? [ApikiS93a] Apiki, Steve "A Unified Ink Standard", BYTE Magazine, September 1993, p. 26 JOT standard for electronic ink storage? [AppleNewton93a] Apple "Apple Newton product", (various) File of references to Apple Newton: August 1993 introduction? Hand-held PDA, resistive touchscreen, scribble/rubout gesture. Includes comparison of Palm PDA and Apple Newton for physical size (Newton too big for hand) [ApplePenLite93a] Apple "Apple PenLite Tablet Computer", (Various) Electronic file of references to Apple PenLite, a pen-computer based on Apple PowerBook Duo from 1992/1993. Project cancelled 1994 in favor of Apple Newton PDA. Most of material in German. See also video "Watch Apple PenLite Video". References obtained October [ApplePenLite93b] Apple "Watch Apple PenLite Video", YouTube.com Video of Apple PenLite pen-computer prototype, circa Mac PowerBook Duo with touchscreen, same UI: touchscreen/stylus replaces mouse as locator device. [ApteA93a] Apte, A. and Kimura, T.D. "A Comparison Study of the Pen and the Mouse in Editing Graphic Diagrams", Proc IEEE 1993 Symposium on Visual Languages, 1993, Bergen, Norway, pp Citrin96 cites for single-stroke graphical editing gestures: comparison of NoteTaker on PenWindows / Windows for Pen Computing without gesture (menu picks only) and the GDE gesture-based graphical editor in GO/Penpoint. Concludes that for drawing, gesture-based editor with a pen/stylus is preferred by users because modes do not have to be changed to a menu. [ApteA93b] Apte, Aajay; Vo, Van; and Kimura, Takayuki Dan "Recognizing Multistroke Geometric Shapes: An Experimental Evaluation", Proc. UIST '83, Atlanta GA, November 3..5, 1993, pp Sketch recognition allowing shapes to be independent of number of strokes e.g. dotted circle and whole circle. Timeout for segmentation increases with number of data points from GO PenPoint touchscreen tablet. [ArmbrusterJL93a] Armbruster, Jeffrey L. "PenRight! Pro SDK 3.1", Windows/DOS Developer's Journal, Vol 4. No. 3, March 1993, pp (Abstract only) PenRight. Pen Computing development environments: PenRight! for Windows review article, 1993 [AscensionTechnology91a] ascension-tech.com "Ascension Technology: The Flock of Birds - Position and Orientation Measurement System, Installation and Operation Guide for Standalone and Multiple Transmitter/Multiple Sensor Configurations", Ascension Technology Corporation, 2004 (first version 1991) 3D/6DOF "tablet" position sensors, used in a number of hand-gesture projects, virtual reality inputs (i.e. sense position and orientation of toy "gun". Multiple small electromagnetic sensors with cables to match one or more multiple base transmitters. Original version of manual [AsherDJ92a] Asher, David J. "Touch Sensor and Controller", United States Patent 5,159,159, October 27, 1992 Pressure/force-sensing tablet with force-sensitive resistive material between two layers. Shows circular/radial conductor patterns as well as X/Y grid array. [AvitzurR92a] Avitzur, R. "Your own handprinting recognition engine", Dr. Dobb's Journal, April 1992, Vol 17 No 4, pp combination Chain-code/zone-based handprinting/handwriting recognizer. Open-source. Csspne.tk blog posting says Avitzur also author of Milo mathematical equation editor for Apple Mach. [BachTW92a] Bach, Thomas William "Device for determining the presence and-or characteristics of an object or a substance", PCT International Patent Application Wo 92/08947, May 29, 1992 Capacitive sensor for sensing internal properties of materials by coupling signal through object, determine either amplitude and/or frequency change. Using compressive membrane, determines pressure, determines shape by distance profile. Refers to sensing near LCD display as touchscreen. [BadlerNI91a] Badler, Norman I. "Human Body Modeling and Simulation", US Army Research Office Report ARO MA, December 1991 Mentions Ascension Technology 6DOF/3D digitizer sensors used in robotic input for model of human body. [BaeckerR91a] Baecker, Ron; Small, Ian; and Mander, Richard "Bringing icons to life", CHI '91 Proc. SIGCHI Conf. 8:26:37 PM]

198 ( ) History of Pen and Gesture Computing on Human Factors in Computing Systems pp. 1-6; also in Baecker95a Toward the Year 2000 Animated icons in GUI: animation can show behavior of tool/operation selected by icon. Example is a pile of documents, top document can be removed from pile. Compare with Freestyle? [BalasubramanianPS93a] Balasubramanian, P.S.; Levy, L.; and Yarm, J. "Image Orientation Sensing and Correction for Notepads", IBM Research Disclosure n , March 01, 1993 Notepad/portable computer with touchscreen, sensors detect rotational orientation of display, adjust user interface for landscape/portrait layout. [BallBR91] Ball, Brian R. "Readings in human-computer interaction: Toward the Year 2000", Morgan Kaufmann Publishers, 1995, ISBN (partial copy) Chapter "Touch, Gesture, and Marking": device independence and virtual devices, equivalence of various pointing/locator devices, mouse/joystick, button/blow-suck tube, touchscreen, etc. [BarlowDC93a] Barlow, Douglas C. "Computer Network Operating with Multilevel Heirarchical Security with Selectable Common Trust Realms and Corresponding Security Protocols", United States Patent 5,204,961, April 20, 1993 Strick hierarchy of trusted systems, systems enforce specific security policies for systems in particular security/trust realms. All inter-system messages sealed (signed/encrypted) and authenticated PKI. [BarrettDM93a] Barrett, David M., McNally, J. Michael, Martin, Patricia A., Huntington, Monathan T. II and Douthart, Robeert M. "Computer with separate display plane and user interface processor", United States Patent 5,260,697, November 9, 1993 Divisional application of UI work by Ward, this one is for the hardware. UI include handwriting recognition bar and simulated devices. Handwriting bar on The Guide at Wang. [BartholomewJ92] Bartholomew, Jim "Microsoft Visual Tools Suite for Windows Ver 1.5", Review of Microsoft development tool suite. Mentions VBX controls added to Control Palette in Visual C++ App Studio or Visual Basic Toolbox, four controls for Windows for Pen Computing Windows for Pen Computing development environments: Visual C++ App Studio graphical environment support VBX controls for Windows for Pen Computing. [BartlettJF92a] Bartlett, Joel F. "Transparent Controls for Interactive graphics", WRL Technical Note TN-30, Digital Western Research Laboratory, July 1992 Shows transparent control panel / menu window with buttons over another image. Uses stippling to get the see-through effect. [BaudelT93a] Baudel, Thomas and Beaudouin-Lafon, Michel "Charade: Remote Control of Objects Using Free- Hand Gestures", CACM, Vol 36, No 7, July 1993, pp Hand-gesture control of presentation graphics using DataGlove: next-chapter gesture is open hand (start) wiping movement of hand (flick gesture to right), close hand. Syntax is Start/Dynamics/End. Gesture recognized by angle/orientation of hand, open and closed fingers and thumb, bendings of fingers. [BelgeM93a] Belge, Matt; Lokuge, Ishantha; and Rivers, David "Back to the Future: A Graphical Layering System Inspired by Transparent Paper", Proc. CHI '93 INTERACT '93 and CHI '93 Conf. Companion on Human Factors in Computing Systems pp Human factors study of transparent/translucent/opaque layers in graphical user interfaces (e.g. CAD GUIs). Users would determine what layer something was on from optical cues, or by moving one layer and seeing which objects moved (because they were in that layer). Transparency/blending done with Gain Momentum GUI system. [BennettWE91] Bennett, William E., Boies, Stephen J., Davies, Anthony R., Etzold, Karl-Friedrich, Rodgers, Todd K. "Optical Stylus and Passive Digitizing Tablet Data Input System", United States Patent 5,051,736, September 24, 1991 Optical stylus digitizer utilizing extremely small patterns within a cell, some similarity to Anoto [BierEA91a] Bier, Eric A. "EmbeddedButtons: Documents as User Interfaces", Proc. UIST '91, Nov 11-13, 1991, pp Arbitrary document elements may behave as buttons, including graphics and text, and document may be control panel for an application. Describes multi-state button. double-click gesture on a button? Button also shows state? Has pop-up menus. [BierEA91b] Bier, Eric A. and Freeman, Steve "MMM: a user interface architecture for shared editors on a single screen", Proc. UIST '91, Nov 11-13, 1991, pp MMM: Multi-Device Multi-User Multi-Editor. Conference editor, everyone uses the same screen: whiteboard with multiple remote pointers, etc. Users can work simultaneously on different parts/attributes of document, can also collaborate: e.g. two-mouse (multi-touch) zoom/pinch gestures. Double select highlighted (overlapping) if two users select same thing or have overlapping selections. Multiple input devices on one machine for multiple users, or remote machines connecting to central server. [BierEA91c] Bier, Eric A. and Pier, Ken "Documents as user interfaces", Proc. CHI '91 SIGCHI Conf. on Human Factors in Computer Systems, pp :26:37 PM]

199 ( ) History of Pen and Gesture Computing Arbitrary document elements may behave as buttons, including graphics and text, Has pop-up menus. Button action is user-specifiable script. Small targets (buttons) targetable using "gravity" snap-to feature of Gargoyle. [BierEA93a] Bier, Eric A.; Stone, Maureen C; Pier, Ken; Fishkin, Ken; Baudel, Thomas; Conway, Matt; Buxton, William; and DeRose, Tony "Toolglass and Magic Lenses: The See-Through Interface", Proc. SIGGRAPH '93, pp Two-handed input (touchscreen, stylus/tablet, trackball and mouse cursor: multitouch?) trackball moves a virtual transparent layer, for transparent objects, such as a transparent selection/framing tool and a mouse click to make an action happen, for example a virtually-transparent color selector or shape selector. dropshadow selector [Blaszczak93] Blaszczak, Mike "Presentations delivered: Mike Blaszczak", Presentations on Windows for Pen Computing in Visual Basic, Windows for Pen Computing development environments: mentions whitepaper presented in 1993/1992 concerning development in Visual Basic and Windows for Pen Computing [BlonderGE92a] Blonder, Greg E. and Boie, Robert A. "Information Processing / Capacitive Moments Sensing for Electronic Paper", United States patent 5,113,041, May 12, 1992 X/Y electrostatic/capacitive grid tablet/transparent touchscreen, uses analog interpolation (zero-crossing) instead of digital/calculated: analog circuitry in stylus looks for zero-crossing of induced signals. Forcedependent capacitor for measuring tip force/pressure (film/sheet on tablet pressed towards lower grid). No reference to Bell Labs 1989/1991 "capacitive moments" report cited in electronic instrument references e.g. Jaffe 93 essay. (Note: Different citations give different title for same authors and patent number.) [BloombergDS91a] Bloomberg, Dan S. "Multiresolution Morphological Approach to Document Image Analysis", ICDAR, Saint-Malo, France, October 1991 Image analysis for extracting text areas from scanned document images. Uses bitblt/raster operations for optimization to word/byte memory boundaries in computer bitmaps. [BloombergDS92a] Bloomberg, Dan S. and Tow, Robert F. "Adaptive Scaling for Decoding Spatially Periodic Selfclocking Glyph Shape Codes", United States Patent 5,091,966, February 25, 1992 DataGlyphs: binary patterns for encoding information on printed paper, glyphs formed by angle of small marks, darkness of marks used for pixels of image instead of dots. Compare with Anoto, other references. [BlySA93a] Bly, Sara A.; Hodges, Jeffrey D.; Kupfer, Michael D.; Lewis, Brian T.; Tallan, Michael L.; and Tom, Stephen B. "Updating local copy of shared data in a collaborative system", United States Patent 5,220,657, June 15, 1993 Project and resource management system [BoltRA92a] Bolt, Richard A. and Herranz, Edward "Two-handed Gesture in Multi-Modal Natural Dialog", Proc. ACM UIST '92, Nov 15-18, 1992, Monterey, CA, pp Discussion of multi-hand gestures in free space by human user subjects: includes rotate gestures for rotating in 3D. Experimental hardware used DataGloves by VPL Research (finger positions) and Polhemus Navation Sciences 6DOF position/orientation sensor. [BonadioAR93a] Bonadio, Allan R. "Apparatus and method for interactively manipulating mathematical equations", United States Patent 5,189,633, February 23, 1993 GUI graphical user interface for mathematical expression parsing and editing. Expressions are typed in, then can be manipulated/evaluated using direct manipulation editor. [BourlandJD91a] Bourland, Joe D; Barbs, Charles F.; Geddes, Leslie, Tacker, Willis A. Jr.; and Graber, George P.; "Pressure Mapping System with Capacitive Measuring pad", United States Patent 5,010,772, April 30, 1991 Multi-point pressure measuring pad for patients on a bed, using array of capacitive sensor nodes formed by grid of conductors with elastomeric rubber material between. Compare with multi-touch [BoyceJ92a] Boyce, Jim "Maximizing Windows 3.1", New Riders Publishing, 1992 (hardcopy book) Multiple color schemes defined in control.ini file of Windows 3. Dynamic themes/scheme management in Windows for color schemes. [BricklinD93a] Bricklin, Dan "JOT defines electronic ink", BYTE Magazine, Vol. 18 No 11., October 1993, page 110 (Abstract only) Short description of JOT standard for electronic ink data for GO/PenPoint and Microsoft Windows for Pen Computing. [BrocklehurstER91a] Brocklehurst, E.R. "The NPL electronic paper project", International Journal of Man-Machine Studies, Vol 34 No 1, January 1991, pp (Abstract only) cross-out gesture [BrocklehurstER91b] Brocklehurst, E.R. and Kenward, P.D. "Preprocessing for Cursive Script Recognition", NPL Report DITC 132/88, 1988 Cited in FordDM91a [BruggeJ92a] Brugge, John "Introduction to PenPoint", MacTECH,The Journal of Macintosh Technology: available 8:26:37 PM]

200 ( ) History of Pen and Gesture Computing at Overview article on PenPoint from about one year after PenPoint OS announced in Apple journal. Detachable (intermittent) networking, gestural support, notes handwriting recognition is limited but not described as a problem in context of operating system. [BryanM93a] Bryan, Marvin "The MacWrite Pro Book", Sybex MacIntosh Library, 1993 (hardcopy book) MacWrite word processing / electronic publishing [BucklandMK92a] Buckland, Michael K. "Emanuel Goldberg, Electronic Document Retrieval, and Vannevar Bush's Memex", J. Am. Soc. for Information Science, Vol 43 No 4, May 1992, pp Vannevar Bush's Memex 1945 likely anticipated as a technical development by 1931 Patent by Emanual Goldberg, and reference would/should have been known. Also mentions 1934 work by Paul Otlet, 1935 by Walter Schuermeyer and electronic document retrieval, others. [BurdeaG92a] Burdea, Grigore; Roskos, Edward; Silver, Deborah; Thibaud, Francois; and Wolpov, Robert "A Distributed Virtual Environment with Dextrous Force Feedback", Proc. Informatique '92, International Conference Interface to Real and Virtual Worlds. pp , Nanterre, France: EC2 Hand controller with two fingers + thumb force feedback. No haptic/tactile feedback, intended to be used with Dataglove for input. [BurskyD92a] Bursky, Dave "Chip Set Simplifies the Design of Pen-Based PCs", Electronic Design, November 2, 1992 PICK (Pen Interface Controller Kit) digitizer interface chip set from TriTech Electronics, to support Scriptel electrostatic digitizer in pen-computing. Refers to changes in pen-position frequency (from 100KHz?) to deal with environmental noise from backlight, etc. [Buxton91b] Buxton, Bill "GEDIT - The Use of Single-Stroke Marks (Marking Menus) in Graphical Interaction", Video posted by Bill Buxton in 2010 on GEDIT, marking menus with single strokes in Menus pop up either on a stationary hold or on a radial stroke ending with a stationary hold. Dwell/press-and-hold: 0:20 stroke, 1:05 hold at end of stroke, 1:27, hold, 1:55 hold and drag. Click/tap gesture known in the art. [Buxton92a] Buxton, W. "The Active Desk", fetched 2011 The Active Desk: posting by Bill Buxton on Xerox research project on surface computing, display and digitizer (Scriptel) in physical desktop, rear projection display. Used in Bricks graspable user interface, collaborative drawing/whiteboard, prototype for VisionMaker Digital Desk. [BuxtonWAS92b] Buxton, William A.S. "Telepresence: Integrating Shared Task and Person Spaces", Proc. Graphics Interface '92, pp Telepresence for videoconferencing, shared drawing space (whiteboard touchscreen). Cites to liveboard, 3D stereo whiteboard, VideoDraw, Videowhiteboard. Importance of human gestures in human communication. In Videowhiteboard (compare with ClearBoard), other participant shown as "shadow" on wall touchscreen. [BuxtonWB93a] Buxton, William "MATE: A Graphical Text Annotation and Editing System Using Digital Ink", University of Toronto, 1993: available at billbuxton.com Video presentation of pen-based text annotation, editing, correction application using tablet, marking menus, and electronic ink. Companion to "A Marking Based Interface for Collaborative Writing". Compare with van Raamsdonk, Slate PenScheduler? [BuxtonWB93b] Buxton, William "The Chameleon: Spatially-Aware Palmtop Computers", University of Toronto, 1993: available at billbuxton.com Video presentation of palm-sized (touchscreen?) device that senses 3D/6D position in a room, user can move device to see 3D presentations of data. Companion to "Situated information spaces" and "Virtual reality for palmtop computers"". Compare with "StarFire" video by Tognazzini? [Byte92a] Baran, Nicholas "Rough Gems: First Pen Systems Show Promise, Lack Refinement", BYTE Magazine, Vol 7 No 4, 1992, pp Reviews of features of 1992 pen computers GridPad, TraveLite from DFM Systems, and Momenta. Compass menu (marking menu) on Momenta. PenDOS, PenWindows, PenPoint. Contemporaneous with release of Windows 3.1 (page 34) which had screen rotation. [Byte92b] Byte Magazine "The Palmtop 386 Is Getting Closer", BYTE Magazine, Vol 7 No 1, 1992, p. 37 C&T announces PC/Chip architecture with 386SX processor. Palmtop / hand-held product plans for PenPoint, GeoWorks, CIC, Microsoft pen computing. PenPoint to run on 386SX-compatible processor. [Byte92c] Byte Magazine "The 1992 Byte Awards of Excellence", BYTE Magazine, Vol 7 No 1, 1992, p. 124 PenPoint operating system given award of excellence by Byte Magazine, true object-oriented operating system. See also page 126, NCR 3125 pen computer, wireless stylus. [Byte93a] Byte Magazine "The 1992 BYTE awards", BYTE Magazine, Vol 18 No 1, 1993 Dynapad pen/tablet computer with PenWindows or PenPoint: two pen apps get awards, PenMagic Numero (spread-sheet) for PenPoint, PenSoft Perspective PIM manager for PenPoint. Same article has tutorial article "Digitizer Renaissance". Mentions PenPoint awards in previous year. 8:26:37 PM]

201 ( ) History of Pen and Gesture Computing [CHI92a] Plaisant, Catherine and Wallace, Daniel "Touchscreen Toggle Design", Proc. CHI '92, May 3-7, 1992, pp Video presentation at CHI 92: "Touchscreen Toggle Design" showing virtual toggle and slider switches: See also -- See also Plaisant92a. Compare to Apple slide switch, Fig 2 bottom left? Also teaches equivalents of slider toggle and lever toggle (traditional toggle switch). [CIC92] CIC Communications Intelligence Corporation "CIC Press Release: CIC PenDOS Handwriter Recognition Systems licensed exclusively to IBM", Software Industry Report: October 19, 1992 CIC licenses PenDOS to IBM as exclusive: apparently does not include handwriting recognition technology per se (later sold as Graffiti 2 after Unistroke case) [CJOT93a] Tam, Cynthia "Technology Update: Visual Keyboards", Canadian Journal of Occupational Therapy CJOT, Vol 60 No 2, June 1993, pp Review of visual keyboard / virtual keyboard / on-screen keyboard offerings for Occupational Therapists. ScreenDoors by Madenta Communications, word-predictor dictionary based on user's word-usage history, automatic punctuation and capitalization. WiViK 2.0 (Windows Visual Keyboard), uses dwell for headpointers to click key, dictionary for word prediction and/or abbreviation expansion: successive expanding quadrant keyboard, similar to marking menus: user points to one quadrant of keyboard, gets expansion of that quadrant divided into four successive quadrants, etc. [CUPID93a] Lynn, M. Stuart (Committee chari); Bradner, Scott; Cowles, Robert; Ferrato, Jim; Hall, Steve; Head, Tom; Hanss, Ted; Knight, Robert; Lynch, Clifford; Margulies, Anne; Resmer, Mark; Sewell, Lawrence; Taylor, Carol M.; Wooden, Jeff; and Worona, Steve "CUPID: Protocols and Services (version 1) An Architectural Overview", Proc. CNI Conference, 1993 Consortium for University Printing and Information Distribution (CUPID) sponsored by Coalition for Networked Information (CNI): experimental system for electronic distribution of documents, using networked (internet) interfaces for protected application. Both "push" and "pull" models for distribution. Documents referred to as Printjobs on servers, access through client programs. Authorization codes (license digital rights management DRM). Part of focus is on networked physical printing with access control. [CaldwellDW92a] Caldwell, David W and Medendorp, Nicholas W. "Touch-sensitive Control Circuit", United States Patent 5,153,572, October 6, 1992 Multiple-touch-pad touch sensor, if there are multiple touches on more than one pad (detected by impedance change) output is inhibited to reject multi-touch input. [CallowayJD92a] Calloway, Jack D.; Holzer, Raymond F;, Matheny, Stephen E.; and Orwig, Gary W. "Electronic Maintenance Support Work Station", United States Patent 5,146,404, September 8, 1992 Compare to Kaplow; Allow user to pick through exploded views of a parts assembly diagram at multiple levels. An item picked in a text list is automatically shown in a graphical representation. [CampWO91a] Camp, William O. Jr. "Tablet Digitizer with untethered stylus", United States Patent 5,072,076, December 10, 1991 piezoelectric/resistive film touch-sensitive digitizer with passive single stylus: two resistive sheets with piezoelelectric film (PVDF polyvinyldene diflouride) between. Force on deformable/flexible films causes piezoelectric voltage between two resistive sheets, voltage sensors at four corner of the sheet. Cites to resistive film tablet digitizers, electrostatic/capacitive grid with battery-powered stylus. [CarrR91a] Carr, Robert and Shafer, Dan "The Power of Penpoint", Addison Wesley, 1991, ISBN: Technical background and information and rationale for design of PenPoint operating system by GO corporation for pen computing, incorporating gestures and handwriting recognition, integrated digitizer tablet and display. Includes sections on applications that do not rely on handwriting recognition. Option sheets, writing pads in UI. User interface of display can changed between landscape and portrait orientation. Page 121 "landscape mode" for screen rotation. [CarrR92a] Carr, Robert; Fraisl, Danile J.; Isaac, Steve; and Ouye, Michael "Computer documents as compound documents in a notebook metaphor", PCT International Patent Application WO 92/08199, 14 May 1992 (Later EP A1?) PenPoint/GO User interface using notebook metaphor instead of desktop metaphor: notebook, virtual bookshelf of applications, indexing, tabs, page number. Documents may be embedded in other documents. [Casio93a] Casio, Inc. "Casio Z-7000 Zoomer product information", Casio, Inc. Casio Z-7000 Zoomer PDA: GEOS Operating system, Handwriting recognition (Graffiti?). File contains additional information on other pen/geos products, comparison to Apple Newton. [CawkellAE93a] Cawkell, A.E. "Encyclopedic Dictionary of Information Technology and Systems", Bawker-Saur, 1993 (hardcopy book) Technical dictionary circa 1993 [Chefalas92] Chefalas, Thomas E.; Fujisake, Tetsunosuke; Kim, Joonki; and Tappert, Charles C. "Robust Prototype 8:26:37 PM]

202 ( ) History of Pen and Gesture Computing Establishment in an On-line Handwriting Recognition System", United States Patent 5,121,441, June 9, 1992 Averaging out the training sets in a trainable recognizer to get a better prototype for elastic matching [ChenM91a] Chen, Michael "Two-dimensional Emulation of Three-dimensional Trackball", United States Patent 5,019,809, May 28, 1991 Simulation of trackball on touchscreen/tablet: movement inside circle rotates trackball, movement outside circle twists trackball. Bill Buxton group at U. Toronto (see simulated devices on a tablet). [ChungCY92a] Chung, Connie Y.; Keen, Robert G.; Raymond, Ozella; Sayder, Jan W.; and Tohlen, Dennis A. "Methods of remapping a keyboard within a computer system", United States Patent 5,086,503, February 4, 1992 Re-map keyboard, show new key assignments on screen. Cites to Kaplow for reconfigurable keyboard. [ClarkMR92a] Clark, Michael R. and Mustafa, Musa "Six Degree of Freedom Graphic Object Controller", United States Patent 5,095,303, March 10, 1992 Mouse 3-D/6-D with finger ring for Z axis, three mutually perpendicular side wheels on mouse for inputting orientation. [ClarkPC93a] Clark, Paul C. and Hoffman, Jance J. "BITS: A Smartcard Protected Operating System", CACM, Vol 37 No 11, November 1994, pp Secure embedded operating system for smartcards: Boot code stored on host, smartcard verifies checksums (signatures?) on load. PKI and certificates on card. Compare with Yee. [CloughWA91] Clough, William A.; Ouelette, Daneil; De La Sablonniere, Serge "Portable Computer with Touch Screen and Computer System Employing Same", PCT International Patent 91/12578, 22 August 1991 Portable slate pen-computer using touch screen (not proximity digitizer): Microslate, Inc. of Canada. [CohenO93a] Cohen, Oryx; Meyer, Shawna; and Nilsen, Erik "Studying the Movement of High Tech Rodentia: Pointing and Dragging", InterCHI '93 Conference on Human Factors in Computing Systems, Amsterdam, 1993, pp Comparison of MousePen, Microtouch UnMouse, Touch Pad, Mouse, trackballs, joystick: UnMouse lower accuracy than touchpad for human use (dragging, pointing): not clear that UnMouse was used in absolute or relative mode; refers to poor placement of barrel buttons on PenMouse stylus [ColbyG91a] Colby, Grace and Scholl, Laura "Transparency and Blur as Selective Cues for Complex Visual Information", SPIE Vol Image Handling and Reproduction System Integration, 1991, pp Use depth cues -- blurring and semi-transparency of overlay plane -- to give user cues about importance of information in complex representation. e.g. In multi-layered map, very important information (name of city) is in sharp focus, relevant but less important information blurred or semi-transparent to varying degree (roads, landsat data, etc.) [ComerfordLD93a] Comberford, Liam D., Ellozy, Hamed A., Jelenek, Frederick, Levy, Stephen E., and Nahamoo, David "Method and Apparatus for Improving the Paper Interface to Computing Systems", United States Patent 5,243,149, September 7, 1993 Electronic clipboard tablet computer with audio recording, optical scanner, digitizing tablet [ComputerBusinessReview92a] Computer Business Review "Pen Announcements", Computer Business Review, Feb 25, 1992, Issule No Slate announces vendors for Day-timer Pen Scheduler PIM personal information management software: NCR, Poqet, Momenta, MicroSlate. Kurta cordless pen/stylus for digitizer, Kurta Lite-Touch pen, PenPoint, PenWindows, Pen-DOS (CIC). SunDisk Flash memory for pen-computing market. [ComputerChronicles93a] The Computer Chronicles "PDA Review, 1993", The Computer Chronicles, 1993 Review of PDA devices, including Apple Newton: Sharp Expert Pad (Newon), Psion 3A, Amstrad pen computer with touchscreen, HP 100LX, Casio Zoomer, AT&T EO 440 Personal Computer. Brief segment with CIC PenDOS at start of video. [ComputerChronicles93b] The Computer Chronicles "DOS 6.2 (PenDOS), 1993", The Computer Chronicles, 1993 Review of DOS 6.2, starts with demonstration of CIC PenDOS pen extensions to DOS on touchscreen. [Computerwoche92a] Computerwoche "Modem und Funktelefon schon integriert: Personal Communicator von EO und AT&T stiehlt Apples Newton die Schau", Computerwoche April 12, 1992 At announcement at at Comdex, PenPoint / EO (manufactured by Matsushita) from GO corporation attracts attention away from Apple Newton: cellular data phone already included in device. [Computerwoche93b] Computerwoche "Pen-computing-Anbieter setzen nicht alle technischen Trends um", Computerwoche V. 22, 1993: available at Comparison of pen-enabled products Windows for Pen-Computing by Microsoft and Pen-DOS from CIC with more radical approach of PenPoint by GO Corporation, with operating system designed from the ground up, but none have met market estimates for success. Mentions that Windows for Pen Computing allows the definition of gestures. CIC Handwriting recognition licensed by IBM as best performing. Also Pen-Right from Grid computing. Microsoft includes support for Visual Basic and C 7.0 MFC. [Computerwoche93c] Computerwoche "Ein kleiner Überblick über die Betriebssysteme für Pen-Computer", 8:26:37 PM]

203 ( ) History of Pen and Gesture Computing Computerwoche 26 Feb 1993: available at Listing of pen-computing operating systems and development environments: PenRight from Grid, PenDOS from CIC, penpoint from GO, and Pen-Windows. Mentions one problem of using mouse-type drivers with pen computing is low resolution, less than that of digitizer. Mentions Pen Pal scripting language. [Computerworld92a] Computerworld "Pen computing sparks interest", Computerworld, October 5, 1992, Vol 24 No 40 IBM OS/2-Pen, Tusk all-terrain Super-Tablet, CIC pendos, Concept Technologies Stroke of the Hand cursive word processor, Rite-Pad DOS pen computer. Go/PenPoint with Inkware Note-Take by Ink Development, Personal Perspective PIM by Pensoft, Penmagig Software The Executive Assitant, Sitka PenCentral connectivity, Notable Technologies PenCross crossword puzzle. [Cornett91] Cornett, Johnny A. and Corbett, James D. "Touchscreen Control Panel with Sliding Touch Control", United States Patent 5,053,758, October 1, 1991 Touchscreen GUI technique, where user slides finger to target, and releases on target: intended to get past inaccuracies/mis-registration of touchscreen film digitizer with display. Compare to Litvin techniques, which are not cited. Contains reference to Minsky gesture article, 1984 [CraneHD93] Crane, Hewitt D. and Rtischev, Dimitry "Pen and voice unite: Adding Pen and Voice Input to Today's User Interfaces Opens the Door for More Natural Communication with Your Computer", BYTE Magazine, Vol 18 No 11, October 1993, pp (Abstract only) compare with Anoto-technology products incorporating voice recording for electronic ink notes? "The strengths of pen and voice technologies complement each other to create a powerful, natural user interface." "Speech to text systems must deal with a wide range of homophones words that sound the same but have different meanings." [Csspne.tk92a] "Interacting with computers through pen gestures", March (?) Spreadsheet with gestures on Apple Newton: mentions Numero by PenMagic, and by Slate / Dan Bricklin. Refers also to pen-computing pen-based extension to Apple MacIntosh, Milo, supporting editing of mathematical expressions. Slate PenApps application builder IDE, PenMagic Numero spreadsheet and database, Pensoft Perspective personal information manager PIM. [CushmanWH91a] Cushman, William H. and Rosenberg, Daniel J. "Human Factors in Product Design", Elsevier Science Publishing Company Inc., 1991, ISBN (hardcopy book) Chapters on Keyboards and Input Devices for Computers, Theory and Design Strategies for User Interface Software. Practical Considerations for the Design of User Interface Software. Brief section on auditory displays (speech output interfaces) [CypherA91a] Cypher, Allen "EAGER: Programming Repetitive Tasks by Example", Proc. CHI 91, New Orleans, April 28-May 2, pp UI detects pattern of repeated action sequences (command input) by user (e.g. moving mail messages to different folders), displays "EAGER" icon in pop-up list, user can select that icon or another. Gestures on buttons? [CypherA91b] Cypher, Allen "EAGER Demo Movie", acypher.com/eager, fetched 2010 Web page describes video (on file) from CHI 91 of Eager, programming by demonstration/example. UI learns repeated action sequences by user (e.g. moving mail messages to different folders), displays "EAGER" icon in pop-up list, user can select that icon or another. Gestures on buttons? [CypherA91c] Cypher, Allen "Video presentation: Eager: programming repetitive tasks by example", Proc. CHI '91, pp Video demonstration of EAGER, programming by demonstration/example: UI learns repeated action sequences by user (e.g. moving mail messages to different folders), displays "EAGER" icon in pop-up list, user can select that icon or another. Gestures on buttons? [CypherA93a] Cypher, Allen, ed. "Watch What I Do: Programming by Demonstration", M.I.T. Press, 1993 (partial copy) Collection of papers on programming by demonstration/programming by example. Co-editors: Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, Alan Turransky [DamoreK93a] Damore, Kelley "Borland C++ 4 adds visual controls", InfoWorld, November 8, 1993, page 19 IDE with object browser, class expert, floating palettes, events. Pen Computing development environment: includes VBX Visual Basic controls: does not mention pen computing, but that was supported already in Borland C [DaoJ91] Dao, James; Foyt, David C.; and Allen, Kenneth R. "Keyless Flat Panel Portable Computer - Computer Aided Notebook", United States Patent 5,049,862, September 17, 1991 Notebook style (fold-out) pen computer, similar in appearance to Refalo. Corded stylus. Two flat panel displays in the two sides of the notebook, each a touchscreen (two digitizers), provision for ring binder at the hinge to hold paper. Rotatable change of orientation to accommodate landscape format, or for left- 8:26:37 PM]

204 ( ) History of Pen and Gesture Computing handed writers. Rotatable notebook computer. [Dauphin92a] Dauphin "The Dauphin 5000 Pen-Based Notebook", Press release, Dauphin Computer, Lombard Illinois Pen-computer slate format using Scriptel electrostatic digitizer. File contains additional materials: The Notebook that Points to the Future [DeBaarDJMJ92a] debaar, Dennis J.M.J.; Foley, James D; and Mullet, Kevin E. "Coupling Application Design and User Interface Design", Proc. CHI '92, May 3-7, 1992, pp UIMS/UIDE with automatic GUI layout: data model is not separate from user interface model, user interface is generated automatically from (enhanced) data model for different UIs. Examples include Open Look, OSF/Motif, Apple, and CUA. [DeWidRD93a] DeWid, R.D. "Scroll control Box", IBM Technical Disclosure Bulletin TDB v36 n , April 1, 1993, pp (partial copy) Rather than tie up user interface display space with scroll bars, have a single scroll widget on screen. [DiffieW92a] Diffie, Whitfield; van Oorschot, Paul C.; Wiener, Michael J. "Authentication and Authenticated Key Exchanges", Designs, Codes and Cryptography, Vol 2, 1992, pp Early description of Diffie-Hellman key exchange used in SSL: station-to-station protocol uses certificates [DigitizerTechnology91] Digitizer Technology Company "User's Guide to the Virtual Tablet Editor", Digitizer Technology Company, Redmond WA, 1991 Graphical editor for defining macros, regions, virtual/simulated devices on digitizing tablet: compare with Kaplow? Related to WinTAB specification? [Discover93a] Discover Magazine "1993 Discover Awards: Computer Hardware and Electronics: The Great Communicator", Discover Magazine, October 1, 1993 GO/PenPoint (EO Personal Communicator) innovation award: cellular phone (digital), fax machine, peninput personal computer. Voice annotations on documents (compare Freestyle?). Instant-on: power up on less than five seconds. [DonoghueK93a] Donoghue, Karen "An Intelligent Sketchpad: A Gestural Language for Denoting Temporal Relations in Dynamic Design", Master's Thesis, Media Arts and Sciences, M.I.T., June 1993 Interactive sketching system using gestures on a touch tablet. Examples are taken from football play diagrams, handmarkings: e.g. draw circle around object and and long arrow line to show how the object (e.g. football player) should move. Pen pressure (tip force) used for visual weight/width/shading, stylus tilt for transparency. Marks/gestures for serif and sans-serif fonts for text editing. Double-ended arrow for zoom/scale. Wacom/ Calcomp Drawingboard, Kurta XGT tilt/pressure digitizer tablets. Karen Donoghue was on Freestyle project at Wang with Ken Knowlton. Bill Buxton as reader for thesis. [DrDobbs93a] Dr. Dobb's Journal "New Products: Lattice Introduces PenBase Database management Library", Dr. Dobb's Journal, April 1, 1993 Database library for use with PenRight for pen computers: compare with PenDBase for PenApps [DrDobbs93b] Schulman, Andrew "Examining the Windows AARD Detection Code - A Serious Message -- And the Code that Product It", Dr. Dobb's Journal, September 1, 1993 AARD code on Windows: reverse-engineering of obfuscated code used in Windows 3.1 beta to detect non- MS-DOS (DR-DOS) and display varying error messages and warnings. Default behavior of WIN.COM was to terminate. [DuwaerAL93a] Duwaer, ARne L. "Data Processing System with a Touch Screen and a Digitizing Tablet, Both Integrated in an Input Device", United States Paten 5,231,381, July 27, 1993 Stylus detector digitizer and finger detector touchscreen on top of each other: refers to ITO for sensors, grid for stylus detector sheet for touch detector, but not specific to any particular digitizer technology, [DysonE91a] Dyson, Esther "GO Corporation: our friend the pen - promoting pen-based input system", Release 1.0: Jan 22, 1991 PenPoint application with intermittent network connections, multiple networks supported, full-peer-level file sharing over network (via TOPS). Longer discussion of PenApps. Penpoint MIL machine interface layer, wireless data modem support, deferred network communications. PenWares: comparison of GO Corporation developer's conference to introduce PenPoint operating system (Bill Campbell), and Microsoft similar conference for Pen Windows the following February 21. (Note: The style, pacing, and features of the Microsoft PenWindows presentation closely paralleled the GO/PenPoint presentation.) [DysonE91b] Dyson, Esther "Component software", Release 1.0: 25 February 1991 Page 22: discussion of component software and component reuse, as in GO/Penpoint and Component WorkShop, ParcPlace systems [DysonE91c] Dyson, Esther "Act III -- Notepads: A New Environment, or a Pen off the old block?", Release 1.0: 25 February 1991 Page 24: PenPoint easier to program for more advanced features than PenWindows. Active Book: PDA based on Smalltalk -- compare with Self and with Newton PDA / NewtonScript? Also mentions PenPal 8:26:37 PM]

205 ( ) History of Pen and Gesture Computing software from PenSoft, and Eden Group VPi386 notepad with PenWindows and touch-screen-enabled AutoCad. [DysonE91d] Dyson, Esther "PenBook: Just the Pages Please", Release 1.0: 30 September 1991 Page 14: PenBook, simple (no editing) e-book / e-publishing application for PenPoint OS, similar to Kindle. Limited text mark-up (white-out, highlighting). [DysonE91e] Dyson, Esther "Smart Recognition: The Brain Behind the Screen", Release 1.0: 31 October 1991 Page 1: PenSieve context recognition by SuperScript: handwriting recognition context word list compared to pick list user interface -- for Pen Windows. Page 4: Paragraph's Parascript/Calligraph: ParaGraph handwriting recognition Page 5: Strategic Position -- Faint Heart Never Won Fair Market (Momenta pen computer), Grid PenRight publicly open APIP [DysonE91f] Dyson, Esther "Power through Scripting", Release 1.0: 31 May 1991, pp. 1-3 Scripting of applications for users to use existing applications/components for additional applications: HP NewWave, Apple Events (compared to Objects), Open Scripting. Mentions PenPoint, no details. Dates direct manipulation to 1980s: word-processor direct move of chunks of text, calendar by typing data into right place. [DysonE92a] Dyson, Esther "The Electronic Frontier: Where Personal Computing Goes from Here", Proc. IFIP 12th World Computer Congress, Madrid, September 1992, pp Pens and GroupWare: Critieus GO PenPoint and Microsoft Windows for Pen Computing (Pen Windows) as too expensive ($2000 in 1992), Cites Pen/GEOS (Graphical Environment Operating Systems) and Palm Computing. Quotes Jeff Hawkings regarding Slate for being cross-platform. Sidekick 2 from Borland. [DysonE92b] Michalski, Jerry "Unified Messaging, Part 1", Release 1.0: 31 December 1992, pp. 1-3 Unified messaging (pre-internet): data, voice, fax, video, , real time or store-and-forward. Discusses difference in nature of interaction in shifting of time, and change of media (phone/video, /voice). Unified messaging means easier to use appropriate medium and form (e.g. send the fax you need to, rather than voice mail because do not know fax number). Discusion of PenPoint/GO "GOnna do it all" GO Message Center. Notable Technologies doing PenPoint messaging apps. Apple Newton messaging. PenPoint as replacement/alternative for desktop, Newton as adjunct to desktop. [DysonE92c] Dyson, Esther "Apple's Newton: Handywidget in Sight", Release 1.0: 15 June 1992 comparison of Apple Newton with speculative features of HandyWidget. Features wireless communications and handwriting recognition for Newton, promise of large third-party applications market, system perpetually upgradeable: comparison with PenPoint. Separate from desktop market. Also reports on internal development history at Apple. [DysonE92d] Dyson, Esther "Context in Context", Release 1.0: 30 November 1992 Niche markets (long end of the tail) too small to be taken over by biggest companies ("Microsoft"): large apps market. Breakdown chart of cellular data markets, GO/PenPoint is only specific example of cellular data device. [DysonE93a] Michalski, Jerry "Directory Services", Release 1.0: 23 April 1993 Directory services, basically unified address book for communications to avoid rampant duplication in diverse systems, Newtons/Zoomers interface to existing directories. PenPoint and Super PIM personal information manager. Overview of Internet and X.400 addressing. [EO93a] EO "The World of Messaging: An Introduction to Personal Communications", EO, Inc., 1993 User documentation provided for EO personal communicator, running PenPoint operating system. Describes , FAX, AT\&T Mail services, Applelink. See also ATTHobbit file (functional device) [EO93b] EO "Lookup Guide to the EO Personal Communicator", EO, Inc., 1993 User documentation provided for EO personal communicator, running PenPoint operating system. Describes basic function of PenPoint and EO "Hobbit" hardware: caret, check/tick, circle, cross out, undo, flick up gestures. Press (hold) gesture for move, tap+press two stroke gesture for copy. Notebook app can be rotated on screen, independently of "desktop". See also ATTHobbit file (functional device), Agulnick file. Marquee animated highlighting for object drag selections. [EO93c] EO "EO Personal Communicator: Applications", EO, Inc., 1993 Brochure on third-party applications for PenPoint/EO: InkWare Notetaker, Longhand by Lexicus, MHS link for GO mail by GO: PenASCII, penhost, Slate PenApps Application Builder, Pensoft Perspective, PenMagic Numero, PenCross [EO93d] EO "Getting Started with your EO Personal Communicator", EO, Inc., 1993 User documentation provided for EO personal communicator, running PenPoint operating system. Marquee animated highlighting for object drag selections. [EO93e] EO/ATT "How to Appease a fat cat FROM a gondola", AT\&T / EO, Inc., 1993 Sales brochure for ATT/EO Hobbit personal communicator pen computer tablet. Shows functional unit (actually display) running PenPoint operating system. Applications include Slate Corporation PenApps (similar to Visual Basic?), Pensoft Corporation Perspective forms/reports/planner, PenMagic Software 8:26:37 PM]

206 ( ) History of Pen and Gesture Computing Numero spreadsheet and LetterExpress letters/fax editing, Ink Development Corporation NoteTaker, fax software. Deskin Research Group Gnosis database and document storage software with remote server storage. [EO93f] EO/ATT "EO Personal Communicator: Video", AT\&T / EO, Inc., 1993 Video marketing presentation for EO personal communicator running PenPoint: shows wireless data, notebook metaphor, and cellphone communications: tablet-sized smartphone with handwriting pencomputing user interface [EO93g] EO/ATT "AT\&T EO Personal Communicator: Third Party Product and Services Catalog", AT\&T / EO, Inc., 1993 PenPoint applications: include aha! InkWriter, Gnosis Mobile Information Manager, InkWare NoteTaker 1.2, Longhand cursive handwriting recognition, wireless/network communications applications, PenApps (Slate application builder), external memory flash [EberhardtSP93a] Eberhardt, Silvio P.; Bernstein, Lynne E.; Coulter, David C.; and Hunckler, Laura A. "OMAR - A haptic display for speech perception by deaf and deaf-blind individuals", Proc. IEEE Virtual Reality Annual International Symposium, September , 1993, Seattle WA, pp Experimental system for haptic display to the deaf of a speakers face or vocal tract, using both positional and vibration actuators. Modeled after Tadoma system of deaf placing hand over speaker's face to feel auxiliary positions of speaking. Suggests even simple haptic feedback (e.g. vibrotactile when a pointer is over a selectable object) would be beneficial. [EdwardsMJ93a] Edwards, Martin J. "Touch sensor array systems and display systems incorporating such", United States Patent 5,194,862, March 16, 1993 Array/matrix of touch sensors (touch switches) for multi-touch input in a graphics tablet: sensing elements integrated with LCD. Addressing technique for dealing with large number of touch sensors: touch sensors are bi-stable (maintain state after touched), so that touched locations in the array can be determined. Mentions capacitive touch sensor grids relying on physical displacement of conductors under force. Active matrix (independent) addressing fabricated using thin-film-transistors (TFT). Cited in Ure Patent as technology for multi-touch. [ElbraT92a] Elbra, Tony "Computer Security Handbook", NCC Blackwell Limited, 1992 (hardcopy book) UK handbook on computer security: physical security, software auditing, simple explanation of networks, rudiments of physical storage of back-ups, etc. [EllozyHA91a] Ellozy, Hamed A; Jeanty, Jenry H; and Tappert, Charles C. "Handwriting recognition employing pairwise discriminant features", United States Patent 5,005,205, April Pairwise analysis of characters against prototypes, then voting, then highest score of the winning pair: appears similar to the Pencept technology (not patented) [ElrodS92a] Elrod, Scott; Richard, Bruce; Gold, Rich; Goldberg, David; Halasz, Frank; Jannsen, William; Lee, David, McCall, Kim; Pedersen, Elin; Pier, Ken; Tang, John; and Welch, Brent "LiveBoard: A Large Interactive Display Supporting Group Meetings, Presentations, and Remove Collaboration", Proc. CHI '92, SIGCHI Conf. on Human Factors in Computing Systems, 1992, pp Multi-user remote whiteboard using rear-projection screen and wireless digitizer pen with four additional buttons (for menus, etc.) Cordless pen digitizer projects light onto screen, detected optically. Sweeping gestures for scrolling and panning, also for turning pages in a slide-show application: tap as gesture? Multitouch: up to three users with three pens at same time, therefore can share the tools. Other references state as multi-user / multi-touch: up to three simultaneous pens/users. [Endo91] Endo, Yasuo; Akimichi, Shinji; Hirose, Tadayoshi; Furukawa, Motonori; Yoshida, Kyoko "Context-Base Input/Output System", United States Patent 5,012,521, April 30, 1991 Gesture recognition user interface: refers to recognition of patterns (sketch recognition?) of elements of on-line drawing (curves, arcs, lines), but emphasis is on gestures: X for erase, clash for cut (of two segments), closed wrap (circle or lasso) to cause recognition of handwriting, wrap/lasso plus arrow for magnify, copy, reduce. No modal input, gestures can be input at any time, compare with Tablet PC gestures. Integrated tablet digitizer and display. [FalkenbergB92a] Falkenberg, Bill; Ford, Tom; Neumann, Janet; and Fenton, Pat "Precise Real-time Kenimatic Differential GPS Using a Cellular Radio Modem", IEEE Position Location and Navigation Symposium PLANS 92, Monterey CA, March 24, 1992 Portable GPS with NovAtel "mini" cellular telephone with data, two GRidPad (touchscreen?) computers. Accuracy better than one meter. [FeinerS91a] Feiner, Steven "Hybrid User Interfaces: Breeding Virtually Bigger Interfaces for Physically Smaller Computers", Proc. UIST '91, Nov 11-13, 1991, pp Combine two head-mounted displays (low resolution and high resolution) and two pointing inputs (low resolution and high resolution: use Polhemus 3-D electromagnetic digitizer to track head). Cites to Sutherland 1965 for head-mounted display. Mentions combination of dataglove and stylus for pointing. 8:26:37 PM]

207 ( ) History of Pen and Gesture Computing [FischerAM91a] Fischer, Addison M. "Public key-signature cryptosystem with enhanced digital signature certification", United States Patent 5,005,200, April 2, 1991 CA generates certificates include authority/authorization/permission/rights attributes, along with public key: examples include dollar signing limit for business, requirement for specific co-signature(s). Single certificate may be signed by multiple CAs. Also mentions signing of other objects and combinations of objects. Storing signature separate from document: allows paper documents to be verified by re-entering data from paper document. [FitzmauriceGW93a] Fitzmaurice, George W. "Situated Information Spaces and Spatially Aware Palmtop Computers", CACM Vol 36 No 7, July 1993, pp Portable/palmtop computers (with touchscreen) instrumented to know nearby objects in physical environment: calendar, office setting, phone, etc. Does not talk about display rotation or orientation landscape/portrait [FitzmauriceGW93b] Fitzmaurice, George W.; Zhai, Shumin; and Chignell, Mark H. "Virtual Reality for Palmtop Computers", ACM Trans. on Information Systems, Vol 11, No 3, July 1993, pp Portable/palmtop computers (with touchscreen) with 3D/6D position sensing, used to show a "virtual environment" as moved around. Alternatively, pie menus for navigation. Chameleon prototype uses video camera feed from camera in front of a regular display to a small hand-held display, with Ascension Technology 3D/6D digitizer. Virtual environment showing floating documents in 3D space, old documents to rear. [FitzpatrickGP93a] Fitzpatrick, G.P.; Haynes, T.R.; and Sterrett, W.R. "Translucent Window Attribute", IBM Technical Disclosure Bulletin, Vol 36 No 06A, June 1993 Translucent windows in windowing system, so that when a window is over another you can still see what is underneath. Translucency /semi-transparent done by muting the color of the underlying window (add grey teint) and/or by blurring slightly. [FlowersDR91a] Flowers, Dale R.; Gallarelli, Pat; Goffinet, Kevin P.; Miles, Anthony W., Rudisill, Charles A.; and Vorhees, Kevin H. "Touch sensing display screen apparatus", United States Patent 5,038,142, August 6, 1991 Touchscreen or tablet measuring force at four corners of a monitor with strain gauges, monitor is mounted so that it can only move in the "Z" direction, and that not by much. [Forcier93] Forcier, Mitchell D. "Script/Binary-encoded Character Processing Method and System with Moving Space Insertion Mode", United States Patent 5,220,649, June 15, 1993 Editing of handwriting/electronic ink without recognition of characters, use of editing gestures distinguished from handwritten character/glyphs based on context: word-wrap, inserting spaces, editing of handwritten lines of text: Compare with van Raamsdonk. Press reports from 1999 of Forcier accusing Microsoft of infringement: see correspondence with Forcier [Forcier93a] Forcier, Mitchell D. "Script/Binary-encoded Character Processing Method and System", United States Patent 5,231,698, July 27, 1993 Editing GUI of handwriting/electronic ink without recognition of characters, use of editing gestures distinguished from handwritten character/glyphs based on context. Editing of electronic ink: word-wrap, inserting spaces, editing of handwritten lines of text Distinguishes handwritten electronic ink from other non-handwriting input during editing Compare with van Raamsdonk. Press reports from 1999 of Forcier accusing Microsoft of infringement [FordDM91a] Ford, David Malcom "On-Line Recognition of Connected Handwriting", Ph.D Thesis, University of Nottingham, U.K., May 1991 Contains review of handwriting recognition techniques for discrete, connected, and cursive writing: primitive sub-strokes, elastic matching, feature matching, Freeman chain codes. Digitizer used was a Pencept PenPad, specifically engineered for use in a handwriting-input system. [Fox92] Fox, Abijah Shawhan; Greanias, Evon Constantine; Kim, Joonki; and Tappert, Charles Carson "A system for automatic adjustment and editing of handwritten text images", European Patent Publication Number B1, Proprietor International Business Machines, Armonk, NY. Stroke segmentation and alignment to a base writing line [FrancikE91a] Franci, Ellen; Rudman, susan Ehrlich; Cooper, Donna; Levine, Stephen "Putting Innovation to Work: Adoption Strategies for Multimedia Communication systems", CACM, Dec. 1991, Vol 34 No 12, pp ; also Baecker95a Towards the Year 2000 Report on adoption plan for Wang Freestyle as pen-based system for cooperative/collaborative work, synchronized voice/electronic-ink annotations. Refers to conversational turns enabled in whiteboarding system. Cites to early resistance to telephone as an electric toy (citations) [FranzPJ93a] Franz, Patrick J. and Straayer, David H. "Integrated Keyboard and Pointing Device System with Automatic Mode Change", United States Patent 5,189,403, February 23, 1993 Keyboard with one isometric (force-sensing) joystick key (e.g. "J" or pointing key), automatically switches to joystick/mouse/pointing mode from keyboard input when user presses longer time on pointing key, 8:26:37 PM]

208 ( ) History of Pen and Gesture Computing switches back based on timing recognition of keystroke input. Compare with press-and-hold? [FrederickV93a] Frederick, Vivian and Yasuda, Phyllis, with Yamada, Judy and Koda, Ann "Using Windows 3.0/3.1", McGraw-Hill, 1993 (hardcopy books) Color and pattern Scheme GUI in Windows 3.0/3.1 [FrickeJ93a] Fricke, Jeorg "Tactile graphic computer screen and input tablet for blind persons using an electrorheological fluid", United States Patent 5,222,895, June 29, 1993 Tactile display for the blind, uses a fluid that changes state in response to a voltage. Intended to display braille. [FriedlandM91] Friedland, Mary "They Made It Happen (Momenta)", Pentop Magazine, Nov Founder Kamran Elahian of Momenta pen-computing company: Portia Isaacson compares MADE (Momenta Application Development Environment) with PenPoint and Pen-Windows. [FritznerC91a] Fritzner, Claus; Nilsen, Leif; and Skomedal, Asmund "Protecting Security Information in Distributed Systems", Proc. IEEE Computer Society Symposium on Research in Security and Privacy, 1991, May , 1991, pp Overview of public key security techniques in computer systems: X.509 certificate, Privilege Attribute Certificate for access control. [FrohlichD93a] Frohlich, David "The History and Future of Direct Manipulation", Hewlett-Packard internal publication, 1991 Broad psychological comparison of direct manipulation GUIs (a la Shneiderman) with virtual partners, virtual reality systems. Cites to Shneiderman for three-point definition of "direct manipulation" [Fujitsu91a] Fujitsu "Fujitsu tablet PC History: Poqet Pad", fetched 2010 PoqetPad, PoqetPad Plus: Early pen computer / touchscreen PDA using NEC V20/V25 (8086 clone) processor. MS-DOS with handwriting recognition, electronic ink capture, signature verification, mouse emulation, pop-up virtual keyboard. NestorWriter recognition software and Nestor PenShell API, PenPal toolkit forms software, HyperPAD scripting language (based on HyperTalk). Half-sized display (640x200), resistive-film touchscreen. Landscape and portrait orientation. File contains additional materials [FullerTH91a] Fuller, Thomas H. Jr. "The Ku