[Kschischang et al, 2001; Taylor et al, 2006; Lorenz, 1963]

Transcription

1

2 observation model g Y(t-1) Y(t) Y(t+1) Z(t-1) Z(t) Z(t+1) dynamical model f [Kschischang et al, 2001; Taylor et al, 2006; Lorenz, 1963] 2

3 n- observation model g Y(t-2) Y(t-1) Y(t) Y(t+1) Z(t-2) dynamical Z(t-1) Z(t) Z(t+1) model f m- [Kschischang et al, 2001] 3

4 n- observation model g Y(t-2) Y(t-1) Y(t) Y(t+1) Z(t-2) dynamical Z(t-1) Z(t) Z(t+1) model f m- p p [Kschischang et al, 2001] 4

5 [LeCun et al, 1998; Mirowski et al, 2007, 2008, 2009]

6 E o (t-1) Y(t-1) observation energy E o (t) Y(t) dynamical energy E d (t) observation parameters W o Z(t-1) dynamical parameters W d Z(t) Learning and inference: deterministic gradient-based EM [Ghahramani & Roweis, 1999] 6

7 [Ghahramani & Roweis, 1999; Ranzato et al, 2007] 7

8 [LeCun et al, 1998b; Ghahramani & Roweis, 1999; Ranzato et al, 2007] 8

9 9

10 [Wierstra et al, 2007] 10

11 Correlation dimension 2.06 Correlation dimension 1.88 [Lorenz, 1963; Mattera et al, 1999] 11

12 [Lendasse et al, 2004] 12

13 [Taylor et al, 2006] 13

14 Original data Reconstruction of missing upper body Results Lower NMSE than nearest neighbors; Inferred smooth, realistic motion Reconstruction of missing left leg 14

15 Original data Reconstruction of missing upper body Results Lower NMSE than nearest neighbors; Inferred smooth, realistic motion Reconstruction of missing left leg 15

16 16

17 1. Barber, D.: Dynamic bayesian networks with deterministic latent tables. In: Advances in Neural Information Processing Systems (2003) 2. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5 (1994) 3. Dempster, A., Laird, N., Rubin, D.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B 39 (1977) 4. Ghahramani, Z., Roweis, S.: Learning nonlinear dynamical systems using an EM algorithm. In: Advances in Neural Information Processing Systems (1999) 5. Kschischang, F., Frey, B., Loeliger, H.-A.: Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory 47 (2001) 6. Ilin, A., Valpola, H., Oja, E.: Nonlinear dynamical Factor Analysis for State Change Detection. IEEE Transactions on Neural Networks 15(3) (2004) 7. Lang, K., Hinton, G.: The development of the time-delay neural network architecture for speech recognition. Technical Report CMU-CS , Carnegie-Mellon University (1988) 8. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, (1998a) 9. LeCun, Y., Bottou, L., Orr, G., Muller, K.: Efficient backprop. In: Orr, G.B., Muller, K.-R. (eds.) NIPS-WS LNCS, vol. 1524, Springer (1998b) 10. Lendasse, A., Oja, E., Simula, O.: Time series prediction competition: The CATS benchmark. In: Proceedings of IEEE International Joint Conference on Neural Networks (2004) 11. Levin, E.: Hidden control neural architecture modeling of nonlinear time-varying systems and its applications. IEEE Transactions on Neural Networks 4 (1993) 12. Lorenz, E.: Deterministic nonperiodic flow. Journal of Atmospheric Sciences 20 (1963) 13. Mattera, D., Haykin, S.: Support vector machines for dynamic reconstruction of a chaotic system. In: Scholkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods: Support Vector Learning, MIT Press (1999) 14. Muller, K., Smola, A., Ratsch, G., Scholkopf, B., Kohlmorgen, J., Vapnik, V.: Using support vector machines for time-series prediction. In: Scholkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods: Support Vector Learning, MIT Press (1999) 15. Sarkka, S., Vehtari, A., Lampinen, J.: Time series prediction by kalman smoother with crossvalidated noise density. In: Proceedings of IEEE International Joint Conference on Neural Networks (2004) 16. Takens, F.: Detecting strange attractors in turbulence. Lecture Notes in Mathematics, vol. 898 (1981) 17. Taylor, G., Hinton, G., Roweis, S.: Modeling human motion using binary latent variables. In: Advances in Neural Information Processing Systems (2006) 18. Wan, E.: Time series prediction by using a connectionist network with internal delay lines. In: Weigend, A.S., Gershenfeld, N.A. (eds.) Time Series Prediction: Forecasting the Future and Understanding the Past, Addison-Wesley (1993) 19. Wan, E., Nelson, A.: Dual kalman filtering methods for nonlinear prediction, estimation, and smoothing. In: Advances in Neural Information Processing Systems (1996) 20. Wang, J., Fleet, D., Hertzmann, A.: Gaussian process dynamical models. In: Advances in Neural Information Processing Systems (2006) 21. Wierstra, D., Gomez, F., Schmidhuber, J.: Modeling systems with internal state using Evolino. In: Proceedings of the 2005 Conference on Genetic and Evolutionary Computation (2005) 22. Williams, R., Zipser, D.: Gradient-based learning algorithms for recurrent networks and their computational complexity. In: Backpropagation: Theory, Architectures and Applications, Lawrence Erlbaum Associates (1995) 17