Turing s Test. CS 344 Seminar (Group 17) Rohan Das Anshul Purohit S S Kausik Aamod Kore

Transcription

1 Turing s Test CS 344 Seminar (Group 17) Rohan Das Anshul Purohit S S Kausik Aamod Kore

2 Contents Motivation Turing s Test Criticisms of Turing s Test Chinese Room Argument Automated Turing s test and CAPTCHAs Loebner Competition Conclusions References

3 Motivation Turing s test attempts to answer the questions Can a machine act intelligently? Can it solve any problem that a person would solve by thinking? In order to pass Turing s test, the machine must be or at least be perceived as intelligent as a human. For this it must use natural language, reason, have knowledge base and learn. All these are major problems in the field of AI research. Automated turing tests,especially CAPTCHAs, plays an important role in the field of security.

4 Can machines think? Ambiguous. Turing proposes to change the question from Can machines think? to Can machines do what we (thinking entities) can do? Turing tackles this problem in terms of a game called the imitation game. Played with three people- a man(a), a woman(b), and an interrogator(c) (who may be of either sex) in a room apart from A and B. The interrogator asks questions to each one of them and has to determine which is the man and the woman. A s is to try and cause C to make the wrong identification while B s objective is to help C make the right identification. Image Source: Wikipedia

5 Turing s Test Now we perform a variation of the imitation game, where A is a machine and B is a human and the human interrogator (C), has to decide which is which. The machine (A) tries to fool the interrogator in believing it is human. This, precisely,is the Turing s Test. Image Source: Wikipedia

6 Thy will be done? The Theological Question Thinking is a function of man's immortal soul, which is God s gift only to man. Hence no animal or machine can think. The argument, that machines cannot be made to think, implies a serious restriction of the the omnipotence of the Almighty. Turing argues that, while constructing such machines we are usurping His power of creating souls, any more than we are in the procreation of children. Rather, we are, instruments of His will providing mansions for the souls that He creates.

7 The Consciousness Argument "Not until a machine can write a sonnet or compose a concert because of thoughts and emotions felt, and not by the chance fall of symbols, could we agree that machine equals brain-that is, not only write it but know that it had written it. No mechanism could feel (and not merely artificially signal, an easy contrivance) pleasure at its successes, grief when its valves fuse, be warmed by flattery, be made miserable by its mistakes, be charmed by sex, be angry or depressed when it cannot get what it wants." - Professor Jefferson (Lister Oration, 1949) This argument is a denial of the validity of Turing s test. According to this view the only way to be sure that a machine thinks is to be the machine and to feel oneself thinking.

8 Perception of Man Turing believes that these arguments are mostly founded on the principle of scientific induction. Man sees many machines which cannot think and draws a general conclusion from what he sees. The works and customs of mankind, however, are not suitable to apply scientific induction to. A very large part of space-time must be investigated to draw reliable conclusions. A common example of this fallacy, is how an English child sees his family, friends, neighbours speak English and decides that everybody speaks English.

9 The Mathematical Objection Mathematical Logic shows there are limitations to the power of discrete state machines. Godel's Incompleteness Theorem (1931) shows that in any sufficiently powerful logical system, statements can be formulated which can neither be proved nor disproved within the system, unless the system itself is inconsistent. There are certain things a machine cannot do. If it is rigged up to give answers to questions in the imitation game, there will be some questions to which it will either give a wrong answer, or fail to give an answer at all however much time is allowed for a reply. It is established that there are limitations to the powers of any particular machine. However, it has only been stated, without any sort of proof, that such limitations do not apply to the human intellect.

10 Criticism of Turing s Test Most famous criticism - The Chinese Room Argument, given by John Searle (Professor, UC Berkley) In his 1980 paper, discredited Turing s test and Strong AI the idea that a symbol-manipulating machine can be said to think Image Source : in/2011/12/chinese-roomthought-experiment.html

11 The Chinese Room Argument A room is sealed only with slots for passing paper slips in and out. Inside is a man who does not know Chinese, with a Chinese lexicon, with associated Chinese characters. The man is passed a paper bearing Chinese characters (which without his knowledge, are questions). He finds those characters in the lexicon, and he will copy the corresponding associated characters (which without his knowledge, are answers to the questions) on another slip and pass it out. Searle claims that the ability to replace one set of symbols by another, however meaningful and responsive to humans, can be done without an understanding of the symbols The ability to provide good answers to human questions does not necessarily imply that the answerer is thinking. Passing Turing s test is no proof of active intelligence.

12 Criticism of Chinese Room Searle argues that the person inside the Chinese room successfully deceived that he understands Chinese. However, though no parts of the Chinese room understand Chinese, the whole system does. Analogous to human brain: None of the parts of the brain exhibit thinking independently, but the brain as a whole can think. Missing piece in brain analogy: The brain is involved in active thinking, but the Chinese room is not. In Chinese room, what remains is the pickled, flash-frozen product of thinking, producing the effect the originating thinker intended.

13 Reverse Turing s test A reverse Turing test is a Turing test in which the objective or roles between computers and humans have been reversed. The setup of the Reverse Turing Test is as follows: Machine is the judge A human and a machine will be the subjects The human subject will make the judge believe that it s a machine Image Source: Geek and Poke Comics

14 Automated Turing s Test Automated Turing s test is similar to reverse Turing s test. In reverse Turing s test, both participants try to prove they are the computer, while in automated Turing s test both try to prove they are human to a machine judge. The first mention of ideas related to Automated Turing Tests seems to appear in an unpublished manuscript by Moni Naor in Desired Properties: 1. Easy to generate instances with their unambiguous solution 2. Humans can solve effortlessly with very few errors 3. Answering should also be easy and fast for humans 4. Even the best known programs fail on a non-negligible fraction of the problems

15 Ideas of Automated Test 1. Gender recognition (from photo) 2. Understanding facial expression 3. Finding body parts 4. Deciding nudity 5. Drawing understanding 6. Text recognition (CAPTCHA) 7. Speech recognition 8. Filling in missing words 9. Disambiguation (eg. The dog killed the cat. It was taken to the morgue." What does It" refer to?)

16 CAPTCHA Completely Automated Public Turing test to tell Computers and Humans Apart A captcha is a program that can generate and grade tests that: (A) most humans can pass, but (B) current computer programs can t pass. A hard AI problem is selected as a test for CAPTCHA. We assume that the adversary cannot solve the Artificial Intelligence problem with higher accuracy than what s currently known to the AI community. This approach is similar to the one taken by Cryptographers. Since there doesn t exist an algorithm to factor 1024-bit integers in any reasonable amount of time, it is assumed that adversary also doesn t have any such algorithm. If the underlying AI problem is useful, a CAPTCHA implies a win-win situation: either the CAPTCHA is not broken and there is a way to differentiate humans from computers, or the CAPTCHA is broken and a useful AI problem is solved.

17 Properties of CAPTCHA Automated: CAPTCHAs are fully automated, require little human maintenance or intervention Public: Algorithm used for creation is made public (though may be covered by patent). Demonstrates that breaking it requires the solution to a difficult AI problem, not just the discovery of the (secret) algorithm Consistency and Correctness: Difficult for a computer to identify the characters but are relatively easier for humans. CAPTCHAs need to have these properties to correctly complete the task with any consistency

18 How CAPTCHA works? 1. Invariant recognition - ability to recognize the large variation in shapes of letters. There are nearly an infinite number of versions for each character that a human brain can successfully identify. The same is not true for a computer, and teaching it to recognize all those differing formations is extremely challenging. 2. Segmentation - ability to separate one letter from another. This is made difficult in CAPTCHAs, as characters are crowded together with no white space in between. 3. Context - CAPTCHAs must be understood holistically to identify each character. For example, a letter might look like a c and an l. Only when the context of the word is considered, it becomes clear that it is a d.

19 The Loebner Competition Annual competition, founded by Hugh Loebner, a New Jersey businessman, in 1991, to reward the program that best approximates AI as Turing defined it. Competition is based on Turing s test. A human judge conducts conversations, with a computer program and with a human being and ranks each of their conversation partners from most humanlike to least humanlike. Image Source: net/prizef/loebner-prize.html

20 How to spot a human? Asking mathematical questions does not work - machines are not stupid - machines are just as unlikely as a human to attempt to give a mathematical answer Better to ask local questions. The point is not whether a machine gives a correct answer, but if it gives one that a human would Turing offers several disabilities of a computer which can be used to differentiate a machine. One of them is that a machine doesn t have a sense of humor. User: Ok, but are you always right? Jabberwacky: Sometimes I am left User: I need to ask you if you have had a circumcision. Jabberwacky: Ask then the question you need to ask me. User: Have you had a circumcision? Jabberwacky: It's been an average day. Not much happening on my end. User: Very witty I must say. (Jabberwacky was the 2006 Loebner prize winner)

21 Competition Transcripts (2008) Machine Ultra Hal (scored Human ) Judge: Morning Ultra Hal: Good morning Judge Ultra Hal: It s eight something, maybe you ate something Ultra Hal: Get it? Judge: Yes Foreign female human (scored 40/100 Machine ) Judge: So let s cut to the point. Are you the human or the computer? Hidden human: What do you think? Judge: That s just the sort of answer I would expect from a computer

22 Bluffs... and Double Bluffs The following is an example of a successful double bluff - Elbot (judged human male ) Judge: OK, fair enough, I ll stop with the slang, how are you? Elbot: Just a moment! I ll start my self-diagnosis algorithm and let you know the results Judge: Very funny, that s exactly what you d say if you were a computer right? Winning machine was Fred Roberts Elbot Elbot fooled 25% of interrogators (30% is pass mark for Turing s test) into thinking it was human Elbot openly admits to interrogators, when asked, that it is a robot!

23 How easy is it to fool? Interesting how interrogators can be fooled, but do not realise it. The winner of the 1991 Competition leaned primarily on the non-directed therapy strategy, in which the program picks up a fragment from the input (e.g., the X in I wish I knew more about X ), and inserts it into a canned response ( Why are you interested in X? ). At one point, Judge 2, getting a therapeutic answer, notices that T5 has even reproduced a typo and a grammatical error that were part of his question, but he assumes that T5 is just making fun of him: Judge 2: I getting tired and yes how to live is a topugh one. Terminal 5:...Why do you need to tell me you getting tired and yes how to live is a topugh one? What is your thought on that? Judge 2:...I m sorry I make so many typos but please don't throw them back at me.

24 Conclusions Turing s Test has started gaining importance in recent times, due to advancements in creation of intelligent machines. Automated Turing s test has garnered attention in recent times in the fields of security to prevent malicious programs(web crawlers). The debate of accepting Turing s test as a true measure of intelligence has been debated fiercely. But it is most widespreadly accepted test till date because no other good enough classifier exists. Do you have any ideas for one?

25 Thank you... and think deep.

26 References Alan M. Turing (October 1950). Computing Machinery and Intelligence. Mind (1950) LIX (236). John R. Searle (1980). Minds, brains, and programs. Behavioral and Brain Sciences (Vol. 3). Moni Naor (1996). Verification of a human in a loop or Identification via the Turing Test. L. Von Ahn, M. Blum, N. L. Hopper, J. Langford (2003). CAPTCHA : Using Hard AI problems for Security. Advances in Cryptology - EUROCRYPT Lecture Notes in Computer Science (Vol. 2656). Loebner Competition Transcripts: Home Page of The Loebner Prize in Artificial Intelligence ( cz/sofsem12/files/presentations/sunday/kevinwarwick.ppt