HIGHER ORDER WAVEFORM SYMMETRY MEASURE AND ITS APPLICATION TO PERIODICITY DETECTORS FOR SPEECH AND SINGING WITH FINE TEMPORAL RESOLUTION

Transcription

1 HIGHER ORDER WAVEFORM SYMMETRY MEASURE AND ITS APPLICATION TO PERIODICITY DETECTORS FOR SPEECH AND SINGING WITH FINE TEMPORAL RESOLUTION Hideki Kawahara, Masanori Morise, Ryuichi Nisimura and Toshio Irino Wakayama University and Ritsumeikan University

2 What I want Parametric representation of expressive voice enabling close-to-natural reproduction and flexible manipulation Example: J-POP (male singer) original version

3 What I want Parametric representation of expressive voice enabling close-to-natural reproduction and flexible manipulation Example: J-POP (male singer) resynthesized version

4 What I want don t Parametric representation of expressive voice enabling close-to-natural reproduction and flexible manipulation Example: J-POP (male singer) dried out version

5 Conclusion Irregularities found in expressive voices consist of very fast modulation of excitation: FM, AM, synchronous and asynchronous Higher-order deviation measure of waveform symmetry provide a method to track such fast modulation of excitation. Stabilized instantaneous frequency calculation can reduce F0 errors significantly with a small degradation in temporal resolution

6 Take home message EGG (Electrogrottograph) signal does not always provide ground truth Fundamental frequency changes slowly is not true Latent variable underlying observed fundamental frequency may be modeled by a slowly changing entity Waveform-symmetry-based method is an efficient and fast F0 measuring device

7 EGG is not almighty EGG speech time (s)

8 glottal closure instance GCI no GCI gender:f talkerid:14 sentenceid:28 reldscrpncy:5.8 % differentiated EGG signal speech signal mismatch EGG signal closed open time (s)

9 amount of mismatch relative mismatch Mismatch is not rare 10% 1% 840 utterances were tested (30 sentences 28 speakers) only 77 (in 840) utterances do not have mismatch record count sorted utterance ID

10 F0 trajectories frequency (Hz) time (ms)

11 Outline Background: STRAIGHT and morphing Excitation source representations Irregularities --> very fast modulation synchronous, asynchronous Waveform-based analyzer Refinement base on a stable representation of instantaneous frequency Demos Forward

12 STRAIGHT Decomposes speech signal into excitation and response Elimination of interferences due to periodic excitation --> response Periodic pulse + colored noise

13 TANDEM-STRAIGHT: natural speech

14 STRAIGHT Decomposes speech signal into excitation and response Elimination of interferences due to periodic excitation --> response Periodic pulse + colored noise Morphing: tool for exploratory research

15 What does morphing do? example:b example:a high-dimensional parameter space

16 What does morphing do? example:b example:a trajectory design high-dimensional parameter space ISCSLP2008, Kunming China, December 2008

17

18

19 STRAIGHT Decomposes speech signal into excitation and response Elimination of interferences due to periodic excitation --> response Periodic pulse + colored noise Morphing: tool for exploratory research

22 Waveform-based analyzer Fundamental frequency is the frequency of the fundamental component --> 1/T0, T0: fundamental interval No ground truth --> back to basic

23 How to select the fundamental component? 0 20 LPF 40 gain (db) frequency (Hz) BPF

24 Architecture LPF 1 Interval measure 31 candidate LPFs from 40 Hz to 800 Hz input signal LPF 2 Interval measure select and estimate by interpolation IF-based refinement F0 with measure Interval LPF N measure

25 Architecture LPF 1 Interval measure 31 candidate LPFs from 40 Hz to 800 Hz input signal LPF 2 Interval measure select and estimate by interpolation IF-based refinement F0 with measure Interval LPF N measure

26 How to measure goodness?

32 How to measure goodness? 0 η E (x, k, f) = α X! 1 1 β w q dβ q (x, k, f) A q K X q K w q =1, dq = d q[k 1] + d q [k]+d q [k + 1] 3 p V (d q )

33 Using window for LPF-FIR 0 Nuttall: #12 in TableII 20 Blackman gain (db) Hanning frequency (Hz)

34 Nuttall #12 in Table II Note: nuttallwin in Matlab is different.

35 Architecture Interval LPF 1 measure input signal LPF 2 Interval measure select and estimate by interpolation IF-based refinement F0 with measure Interval LPF N measure

36 Architecture Interval LPF 1 measure input signal LPF 2 Interval measure select and estimate by interpolation IF-based refinement F0 with measure Interval LPF N measure

37 F0 candidates Peak interval Symmetry measure

38 10 0 w: alph: beta: noise noise+periodic pulse 10 1 probability 10 2 Interspeech signal noise composite measure

39 10 0 w: alph:4 beta:4 noise noise+periodic pulse 10 1 probability Interspeech ICASSP signal noise composite measure

40 Modulation transfer function MTF of Zx, XSX, YIN and SWIPE XSX interval-based refined gain (db) SWIPE YIN modulation frequency (Hz)

42 Stabilized instantaneous frequency

43 Derivation-3 Averaged instantaneous frequency Power weighted average Note! Denominator is TANDEM spectrum

44 Derivation-5 Numerators Substitution and simplification Squared terms vanish

45 F0 refinement 1% rms error interval-based F0 refined F0

47 Waveform and F0 trajectories 2 plain 1 0 expressive time (ms) 350 frequency (Hz) time (ms)

50 F0 trajectories frequency (Hz) time (ms)

51 Spectrum of F0 trajectories relative level (db) modulation frequency (Hz)

53 Spectral envelope plain performance expressive performance

54 Demos J-POP Original Re-synthesis Re-synthesis (F0 modulation suppressed) Re-synthesis (Spectral modulation suppressed) Re-synthesis (Spectral shape equalized) Re-synthesis (All feature suppressed)

62 Demos J-POP Original Re-synthesis Re-synthesis (F0 modulation suppressed) Re-synthesis (Spectral modulation suppressed) Re-synthesis (Spectral shape equalized) Re-synthesis (All feature suppressed) Expressiveness disappears

63 Conclusion Irregularities found in expressive voices consist of very fast modulation of excitation: FM, AM, synchronous and asynchronous Higher-order deviation measure of waveform symmetry provide a method to track such fast modulation of excitation. Stabilized instantaneous frequency calculation can reduce F0 errors significantly with a small degradation in temporal resolution

64 Take home message EGG (Electrogrottograph) signal does not always provide ground truth Fundamental frequency changes slowly is not true Latent variable underlying observed fundamental frequency may be modeled by a slowly changing entity Waveform-symmetry-based method is an efficient and fast F0 measuring device

65 Take home message EGG (Electrogrottograph) signal does not always provide ground truth Thank you! Q&A Fundamental frequency changes slowly is not true Latent variable underlying observed fundamental frequency may be modeled by a slowly changing entity Waveform-symmetry-based method is an efficient and fast F0 measuring device

66 Derivation-1 Flanagan s equation No need of inverse function

67 Derivation-2 Flanagan s equation Simplification and notation =

68 Derivation-3 Averaged instantaneous frequency Power weighted average Note! Denominator is TANDEM spectrum

69 Derivation-4 Averaged instantaneous frequency Power weighted average Numerator: sum of each numerator

70 Derivation-5 Numerators Substitution and simplification Squared terms vanish

71 Derivation-6 TANDEM trick Independent on time This term should be eliminted = TANDEM trick

72 waveform 1996 ASJ annual meeting power (db) /export/hakone/users/kawahara/kawaharactrl/hb1vy100.raw 25 Nov 96 10:12: x 10 4 Amp Headphone x 10 4 frequency (Hz) 10 0 Pitch controller Fndmntlnss F0pwr(dB) channel # x x x 10 4 Workstation Multi channel A/D Mixer Mic Polygraph Photosensor time (ms) x 10 4 F0 with heart beat sync. (a) F0 modulation coherency (a) F0 (Hz) ti ( ) f i i iti time (ms) coherency modulation frequency (Hz)

73