Sigmoid Functions and Their Usage in Artificial Neural Networks

Transcription

1 Sigmoid Functions and Thir Usag in Artificial Nural Ntworks Taskin Kocak School of Elctrical Enginring and Computr Scinc Applications of Calculus II: Invrs Functions

2 Eampl problm Calculus Topic: Invrs functions Sction 7.6 #9: Prov idntity tanh tanh 2

3 Eampl problm (cont.) tanh tanh?? tanh? a) sinh b) cosh sinh c) sinh cosh d)sinh cosh

4 Eampl problm (cont.) cosh sinh cosh sinh tanh tanh

5 Eampl problm (cont.) cosh sinh cosh sinh tanh tanh 2 2 2

6 Larning Objctivs. Dtrmin th rlationships btwn th biological and artificial nural ntworks 2. Usag of artificial nural ntworks Th OR ampl 3. Th classic XOR problm

7 Sigmoid functions ( t) t A moid function producs a curv with an S shap. Th ampl moid function shown on th lft is a spcial cas of th logistic function, which modls th growth of som st.

8 Sigmoid, hyprbolic functions, and nural ntworks In gnral, a moid function is ral-valud and diffrntiabl, having a non-ngativ or non-positiv first drivativ, on local minimum, and on local maimum. Th logistic moid function is rlatd to th hyprbolic tangnt as follows 2( ) 2 tanh 2

9 Sigmoid, hyprbolic functions, and nural ntworks Sigmoid functions ar oftn usd in artificial nural ntworks to introduc nonlinarity in th modl. A nural ntwork lmnt computs a linar combination of its input nals, and applis a moid function to th rsult. A rason for its popularity in nural ntworks is bcaus th moid function satisfis a proprty btwn th drivativ and itslf such that it is computationally asy to prform. d dt ( t) ( t)( ( t)) Drivativs of th moid function ar usually mployd in larning algorithms.

10 Artificial nural ntworks: Motivation. Pattrn classification 2. Clustring 3. Function approimation

11 Artificial nural ntworks: Motivation Optimization Travling salsman problm Start from city A and visit all th citis. What s th shortst path? A E a) A-C-B-D-E-A C b) A-E-B-D-C-A D B c) A-B-C-D-E-A d) A-D-C-E-B-A ) A-C-E-D-B-A

12 Artificial nural ntworks: Motivation 4. Prdiction / forcasting 5. Optimization 6. Contnt addrssabl mmory

13 Biological nuron A nuron (or nrv cll) is a spcial biological cll that procsss information. It is composd of a cll body, or soma, and two typs of out-raching tr-lik branchs: th aon and th dndrits. Th cll body has a nuclus that contains information. A nuron rcivs nals (impulss) from othr nurons through its dndrits (rcivrs) and transmits nals gnratd by its cll body along th aon (transmittr).

14 Artificial nural ntworks Inspird by biological nural ntworks, artificial nural ntworks ar massiv paralll computing systms consisting of an trmly larg numbr of simpl procssors with many intrconnctions. McCulloch and Pitts proposd a binary thrshold unit as a computational modl for an artificial nuron.

15 Activation function - Unit stp function Activation function: A function usd to transform th activation lvl of nuron (wightd sum of inputs) to an output nal. Unit stp function is on of th activation functions Θ N j j j T I w y > < ) ( t t t t if if t t u

16 Assssmnt of Larning Objctiv # Similaritis btwn th biological and artificial nural ntworks Connction wights in ANN rprsnt which biological structur? a) Cll body b) Nuclus c) Aon d) Dndrits ) Synaps

17 Usag of artificial nural ntworks Th OR ampl w will utiliz th McCulloch-Pitt modl to train a nural ntwork to larn th logic OR function. Th OR function w will us is a two-input binary OR function givn in Tabl. Tabl : OR function I I 2 Output ? 0? a) 0 b)

18 Th OR ampl First, w will us on nuron with two inputs. Not that th inputs ar givn qual wights by asning th wights (w s) to. Th thrshold, T, is st to 0 in this ampl. W calculat th output as follows: ) Comput th total wightd inputs X 2 i I i w i XI w I 2 w 2 I I 2 I I 2

19 Th OR ampl 2) Calculat th output using th logistic moid activation function O ( X T ) ( X ) X Now, lt s try it for th inputs givn in Tabl. For I 0 and I 2 0; X0, O 0 0.5

20 Th OR ampl (cont.) For I 0 and I 2, and I and I 2 0; X, O For I and I 2 ; X2, O For all cass th rsults match with Tabl assuming that 0.5 and blow ar considrd as 0 and abov as.

21 Assssmnt of Larning Objctiv #2. (Two minut discussion) If th wights wr 0.5 rathr than, will th ntwork still function lik OR? a) Ys b) No

22 Assssmnt of Larning Objctiv #2 2. (5-minut papr) In groups of two studnts, discuss whthr th sam ntwork can b usd to larn th AND function? (Hint: You may chang th thrshold(0.5) if ncssary) Tabl 2: AND function I I 2 Output ? 0? a) 0 b)

23 Th classic XOR problm Tabl 3: XOR function I I 2 Output If w us th sam on-nuron modl to larn th XOR (clusiv or) function, th modl will fail. Th first thr cass will produc corrct rsults; howvr, th last cas will produc, which is not corrct. 0

24 Th classic XOR problm (cont.) Th solution is to add a middl (hiddn in ANN trminology) layr btwn th inputs and th output nuron Choos th wights ww2w2w22. Us a diffrnt moid function, which is givn with a crtain thrshold for ach nuron: ( ) ( ) ( ) H H 2 O ( 0.5) (.5) ( 0.2) Confirm by calculating th nuron outputs for ach possibl input combinations that this nural ntwork is indd functioning lik an XOR. (Hint: Th output qual or blow 0.5 is considrd 0, othrwis )

25 Nuron calculation I I2 XOR X H H2 O Out ww2w2w22 ( ) ( ) ( ) H H 2 O ( 0.5) (.5) ( 0.2) H H H (0) () ( (2) (00.5) 0.5) (20.5)

26 Nuron calculation (2) I I2 XOR X H H2 O Out ww2w2w22 ( ) ( ) ( ) H H 2 O ( 0.5) (.5) ( 0.2) H 2 H 2 H 2 (0) () ( (2) (0.5).5) (2.5)

27 Nuron calculation (3) ww2w2w22 I I2 XOR X H H2 O Out ( ) ( ) ( ) H H 2 O ( 0.5) (.5) ( 0.2) O O ( H 2 H) ( H 2 H) O O (0.95) (0.2450) ( ) ( )

28 Nuron calculation (4) I I2 XOR X H H2 O Out ww2w2w22 ( ) ( ) ( ) H H 2 O ( 0.5) (.5) ( 0.2) Assuming that 0.5 and blow ar considrd as 0 and abov as.

29 Eampl applications