Perceptual experiments sir-skur-spur-stir

Transcription

1 Perceptual experiments sir-skur-spur-stir Amy Beeston & Guy Brown 19 May 21

2 1 Introduction 2 Experiment 1: cutoff Set up Results 3 Experiment 2: reverse Set up Results 4 Discussion

3 Introduction introduction

4 Introduction Background Based closely on Watkins sir-stir paradigm Gather human data for ASR comparison with/without constancy model Investigate effect of reverberation on stop consonants esp. place of articulation Replicate compensation for reverberation in another lab with naturalistic speech, not interpolated stimuli with further unvoiced stop consonants {k,p,t}

5 Introduction Comparison with Watkins sir-stir work Similarities Two experiments (works in progress) cutoff: frequency effects Watkins and Makin, JASA 27 etc. reverse: time-direction effects Watkins, JASA 25, experiment 5

6 Introduction Comparison with Watkins sir-stir work Differences Listener data consonant confusions (not category boundary shifts) #()*+,'!"#$!%&#$!'&#$!(#$!"#$%&#"#'!"#$!%&#$!'&#$!(#$ percentage correct relative information transferred something else?

7 Introduction % correct and relative information transferred (RIT) m = RIT (m) = 1 % correct(m) = 1 m = RIT (m) = % correct(m) = 25 RIT reflects information about pattern of errors reflects complexity of task - useful for ASR - different sized vocabularies OK RIT = H(X : Y ) / H(X ) H(X : Y ) is the mutual information of X and Y H(X ) is the self-information (entropy) of X Ref: Smith (199)

8 Experiment 1 cutoff Set up cutoff experiment 1

9 Experiment 1 cutoff Set up cutoff experiment Aim: find appropriate parameter set for future experiments should allow effect of reverberation on test word compensation due to reverberation on context Prediction: Extreme low-pass filtering increases misclassification rate also blocks compensation for reverberation

10 Experiment 1 cutoff Set up Stimuli (cutoff) 16 stimuli = 2 talkers 4 words 4 distances 5 cutoffs 8 Articulation Index Corpus utterances 2 talkers, 4 test words {sir, skur, spur, stir} 4 reverberation conditions L-shaped room {near-near, near-far, far-near, far-far} 5 low-pass filter cutoff frequencies 8th order Butterworth {1, 15, 2, 3, 4} Hz Each utterance once to each listener 1 group of 2 subjects

11 Experiment 1 cutoff Results and analysis Results (cutoff) i. percentage error 7 6 lp@1 7 6 lp@ lp@2 7 6 lp@3 7 6 lp@4 test.32 m test 1 m percentage error

12 Experiment 1 cutoff Results and analysis ANOVA (cutoff) i. percentage correct 3-way repeated measures, all within-subject factors Independent variables test-word distance (2 levels) context distance (2 levels) low-pass filter cutoff frequency (5 levels) Dependent variable percentage correct

13 Experiment 1 cutoff Results and analysis ANOVA (cutoff) results i. percentage correct Significant main effects test F (1, 19) = 79.28, p <.1 cutoff F (4, 76) = 24.48, ɛ HF =.7, p <.1 Significant interactions test context F (1, 19) = 8.47, p <.1 context cutoff F (4, 76) = 4.227, ɛ HF =.9, p <.1 No other significant F -ratios

14 Experiment 1 cutoff Results and analysis Results (cutoff) ii. relative information transferred.6.5 lp@1.6.5 lp@ lp@2.6.5 lp@3.6.5 lp@4 test.32 m test 1 m Mean(RIT)

15 Experiment 1 cutoff Results and analysis ANOVA (cutoff) ii. relative information transferred 3-way repeated measures, all within-subject factors Independent variables test-word distance (2 levels) context distance (2 levels) low-pass filter cutoff frequency (5 levels) Dependent variable relative information transferred

16 Experiment 1 cutoff Results and analysis ANOVA (cutoff) results ii. relative information transferred Significant main effects test F (1, 19) = 59.27, p <.1 cutoff F (4, 76) = 9.19, ɛ HF =.96, p <.1 Significant interactions context cutoff F (4, 76) = 2.593, ɛ HF = 1., p <.5 no other significant F -ratios no significant interaction of test context by this measure

17 Experiment 1 cutoff Results and analysis Conclusion (cutoff) Interim conclusion: Compensation replicated best at 3 and 4 khz cutoff conditions Use 4 khz cutoff frequency for future experiments

18 Experiment 2 reverse Set up reverse experiment 2

19 Experiment 2 reverse Set up Stimuli (reverse) 128 stimuli = 2 talkers 4 words 4 distances 4 contexts Articulation Index Corpus 2 talkers, 4 test words {sir, skur, spur, stir} Everything low-pass filtered 8th order Butterworth, cutoff at 4 khz 4 reverberation conditions L-shaped room {near-near, near-far, far-near, far-far} 4 preceding context conditions {forward, reverse} speech {forward, reverse} reverb Each utterance once to each listener 48 subjects = 3 groups of 16

20 Experiment 2 reverse Set up Stimuli (reverse)!"#$%$!"&$%$'$%$!"($!"#$%#"!"&'("!"#$%$!"&$%$'$%$!"!"#$%$!"&$%$'$%$!"($!"#$%#"!"&'("!"#$%$!"&$%$'$%$!"($!"#$%#"!"&'("!"#$%$!"&$%$'$%$!"!"#$%$!"&$%$'$%$!"($!"#$%#"!"&'("!"#$%$!"&$%$'$%$!" Forward reverb cases: context reverb overlaps test word Reverse reverb cases: reverb during test word does not vary with context distance nn=fn, nf=ff!"#$%$!"&$%$'$%$!"($!"#$%$!"&$%$'$%$!"

21 Experiment 2 reverse Results and analysis Results (reverse) i. percentage correct forward speech forward reverberation reverse speech forward reverberation forward speech reverse reverberation reverse speech reverse reverberation test.32 m test 1 m percentage error

22 Experiment 2 reverse Results and analysis Results (reverse) ii. relative information transferred forward speech forward reverberation reverse speech forward reverberation forward speech reverse reverberation reverse speech reverse reverberation test.32 m test 1 m Mean(RIT)

23 Experiment 2 reverse Results and analysis ANOVA (reverse) 4-way repeated measures, all within-subject factors Independent variables test-word distance (2 levels) context distance (2 levels) speech direction (2 levels) reverberation direction (2 levels) Dependent variable i percentage correct ii. relative information transferred

24 Experiment 2 reverse Results and analysis ANOVA (reverse) results Significant main effects i. % correct: test F (1, 47) = 24., p <.1 ii. RIT: test F (1, 47) = 189.5, p <.1 ii. RIT: context F (1, 47) = 5.7, p <.5 Significant interactions i. % correct: test context F (1, 47) = 4.71, p <.5 ii. RIT: context test F (1, 47) = 7.9, p <.1 No other significant F -ratios

25 Experiment 2 reverse Results and analysis ANOVA (reverse) significance per speech & reverb direction fwd fwd rev fwd fwd rev rev rev speech reverb speech reverb speech reverb speech reverb % RIT % RIT % RIT % RIT C nearly yes no no no no no no T yes yes yes yes yes yes yes yes C T yes yes no nearly no no no no

26 Experiment 2 reverse Results and analysis Conclusion (reverse) Interim conclusion: Fwd-fwd case shows typical compensation pattern Reverse reverberation seems to remove main effect of context-distance But... choice of dependent variable influences results considerably

27 Discussion discussion

28 Discussion Confusion Differentiating error patterns m = m = m = m = RIT (m) = 1 % correct(m) = 1 FP sir = RIT (m) = % correct(m) = 25 FP sir = 15 RIT (m) =.19 % correct(m) = FP sir = 15 RIT (m) =.192 % correct(m) = FP sir = 45

29 Discussion Confusion Differentiating error patterns m = RIT (m) =.19 % correct(m) = FP sir = 15 m = RIT (m) =.192 % correct(m) = FP sir = 45

30 Discussion Confusion Receiver operating characteristic (ROC) sir = TP FN FN FN FP TN TN TN FP TN TN TN FP TN TN TN skur = TN FP TN TN FN TP FN FN TN FP TN TN TN FP TN TN spur = TN TN FP TN TN TN FP TN FN FN TP FN TN TN FP TN stir = TN TN TN FP TN TN TN FP TN TN TN FP FN FN FN TP

31 Discussion Confusion Confusions (cutoff) sir skur spur stir sir 18 2 skur spur stir sir skur spur stir sir skur 16 4 spur stir 1 sir skur spur stir sir 19 1 skur 2 spur stir 2

32 Discussion Confusion Confusions (reverse) sir skur spur stir sir skur spur stir 13 2 sir skur spur stir sir 51 9 skur spur stir 4 2 sir skur spur stir sir skur 1 59 spur 6 stir 2 58

33 Discussion False negatives Word-by-word (cutoff) Hz i. False negatives.8.7 sir.8.7 skur.8.7 spur.8.7 stir test 1 m test.32 m # occurrences missed (false negatives)

34 Discussion False negatives ANOVA (cutoff) Hz i. False negatives Independent variables (levels): context (2), test (2), word (4) Dependent variable: # false negative responses Significant main effects context F (1, 47) = 9.67, p <.5 test F (1, 47) = 21.8, p <.1 word F (3, 141) = 42.17, ɛ HF =.44, p <.1 Significant interactions context test F (1, 47) = 8.32, p <.1 test word F (3, 141) = 2.82, ɛ HF =.81, p <.5

35 Discussion False negatives Word-by-word (reverse) fwd-fwd i. False negatives.6 forward speech forward reverberation sir.6 forward speech forward reverberation skur.6 forward speech forward reverberation spur.6 forward speech forward reverberation stir test 1 m test.32 m # occurrences missed (false negatives)

36 Discussion False negatives ANOVA (reverse) fwd-fwd i. False negatives Independent variables (levels): context (2), test (2), word (4) Dependent variable: # false negative responses Significant main effect test F (1, 47) = 61.74, p <.1 Significant interactions context test F (1, 47) = 4.14, p <.5 test word F (3, 141) = 2.82, ɛ HF = 1., p <.5

37 Discussion Sir responses Word-by-word (cutoff) Hz ii. Sir responses 1 lp@4 sir.6 lp@4 skur.6 lp@4 spur.6 lp@4 stir test 1 m test.32 m # sir responses

38 Discussion Sir responses ANOVA (cutoff) Hz ii. Sir responses Independent variables (levels): context (2), test (2), word (4) Dependent variable: # sir responses Significant main effects context F (1, 47) = 13.64, p <.1 test F (1, 47) = 1.422, p <.1 word F (3, 141) = 479.1, ɛ HF =.87, p <.1 Significant interactions context test F (1, 47) = 11.81, p <.1 test word F (3, 141) = 7.28, ɛ HF =.85, p <.1

39 Discussion Sir responses Word-by-word (reverse) fwd-fwd ii. Sir responses 1.2 forward speech forward reverberation sir.45.4 forward speech forward reverberation skur.45.4 forward speech forward reverberation spur.45.4 forward speech forward reverberation stir test 1 m test.32 m # sir responses

40 Discussion Sir responses ANOVA (reverse), fwd-fwd ii. Sir responses Independent variables (levels): context (2), test (2), word (4) Dependent variable: # sir responses Significant main effects context F (1, 47) = 7.96, p <.1 test F (1, 47) = 7.3, p <.5 word F (3, 141) = 74.64, ɛ HF =.99, p <.1 Significant interactions context test F (1, 47) = 8.44, p <.1 test word F (3, 141) = 7.9, ɛ HF =.7, p <.1

41 Discussion Sir responses Recap Much work to do on analysis of current results Future experiments to be designed with ASR experiments in mind (esp. to help tune constancy model)

42 Discussion Thanks...

43 Discussion

44 Appendix extras

45 Appendix 5 Appendix References Stimuli creation Stimuli partitioning details Additional results

46 Appendix References Further reading A.M. Smith. On the use of the relative information transmitted (RIT) measure for the assessment of performance in the evaluation of automated speech recognition (ASR) devices. In Australian International Conference on Speech Science and Technology, pages , 199. A.J. Watkins. Perceptual compensation for effects of reverberation in speech identification. J. Acoust. Soc. Am., 118(1): , 25. A.J. Watkins and S.J. Makin. Steady-spectrum contexts and perceptual compensation for reverberation in speech identication. J. Acoust. Soc. Am., 121(1): , 27. A.J. Watkins and S.J. Makin. Perceptual compensation for reverberation in speech identication: Effects of single-band, multiple-band and wideband noise contexts. Acta. Acust. United Ac., 93:43 41, 27. J. Wright, Articulation Index. Linguistic Data Consortium, Philadelphia, 25.

47 Appendix Stimuli creation Articulation Index Corpus Talker f11 m12 f13 m14 f15 f16 m17 f18 f19 m11 m111 m112 f113 m114 m115 m116 m117 m118 f119 m12 sir they recognize sir entirely anyone detect sir evenly you utter sir more noone see sir today you pronounce sir easily we notice sir sometime I echo sir today people watch sir clearly we show sir tenth you ponder sir first we notice sir seventh I echo sir happily noone suggest sir steadily everyone notice sir anyway I evoke sir precisely people study sir only everyone study sir sixth they read sir properly they see sir easily people note sir typically Talker f11 m12 f13 m14 f15 f16 m17 f18 f19 m11 m111 m112 f113 m114 m115 m116 m117 m118 f119 m12 skur everyone attempt skur tenth someone record skur entirely everyone distinguish skur sometime noone remember skur third noone study skur neatly someone write skur precisely someone imagine skur precisely noone write skur second someone show skur fifth we imagine skur gladly I report skur nicely I think skur first you study skur daily everyone describe skur monthly noone echo skur today I repeat skur surely they distinguish skur wisely someone say skur fifth we sense skur twice people speak skur eighth

48 Appendix Stimuli creation Articulation Index Corpus Talker f11 m12 f13 m14 f15 f16 m17 f18 f19 m11 m111 m112 f113 m114 m115 m116 m117 m118 f119 m12 spur I use spur fluently everyone perceive spur properly we think spur fourth people ponder spur nicely people saw spur nicely we note spur properly they watch spur only I distinguish spur usually someone remember spur easily someone repeat spur anyway everyone propose spur happily they think spur entirely noone hear spur monthly we speak spur surely people echo spur ninth everyone thinks spur fluently anyone prompt spur easily they speak spur seventh someone witness spur now noone watch spur happily Talker f11 m12 f13 m14 f15 f16 m17 f18 f19 m11 m111 m112 f113 m114 m115 m116 m117 m118 f119 m12 stir noone check stir eighth people determine stir ninth they imagine stir surely we determine stir surely they review stir gladly people saw stir steadily I remember stir surely I use stir neatly I use stir wisely we view stir ninth people ponder stir second I evoke stir precisely I read stir second they said stir wisely I echo stir precisely noone report stir well everyone view stir neatly I imagine stir daily you understand stir sixth they sense stir gladly

49 Appendix Stimuli creation Phonetic transcription

50 Appendix Stimuli creation Convolution with Reading BRIRs for L-shaped room

51 Appendix Stimuli partitioning details Partitioning (cutoff) Each AIC utterance presented once only to each listener 2 conditions are tested 4 reverb distances 5 filter cutoffs 16 stimuli partitioned between 2 listeners 1 listener gets 8 utterances 4 utterances at each of 2 conditions Even partitioning 1 word tested at each of 2 conditions

52 Appendix Stimuli partitioning details Presentation (cutoff) Listener seated in a sound-attenuating booth Monaural presentation (left ear) Familiarisation with interface 4 buttons, labelled {sir, skur, spur, stir} Click one button for each trial heard 1 group of 2 listeners age 2-5, both native-english and non

53 Appendix Stimuli partitioning details Partitioning (reverse) Each AIC utterance presented once only to each listener 16 conditions are tested 4 reverb distances 4 preceding context directions 128 stimuli partitioned between 16 listeners 1 listener gets 8 utterances 5 utterances at each of 16 conditions Uneven partitioning 3 words tested once in 16 conditions, 1 word tested twice

54 Appendix Stimuli partitioning details Presentation (reverse) Listener seated in a sound-attenuating booth Monaural presentation (left ear) Familiarisation with interface 4 buttons, labelled {sir, skur, spur, stir} Click one button for each trial heard 48 subjects = 3 groups of 16 listeners age 2-5, both native-english and non

55 Appendix Additional results Cutoff results iii. Pooled (RIT) test.32 m test 1 m Pooled(RIT)

56 Appendix Additional results Reverse results iii. Pooled (RIT).6 forward speech forward reverberation.6 reverse speech forward reverberation.6 forward speech reverse reverberation.6 reverse speech reverse reverberation test.32 m test 1 m Pooled(RIT)