Adaptive testing for video quality assessment. Vlado Menkovski, MSc Eindhoven University of Technology, NL

Size: px

Start display at page:

Download "Adaptive testing for video quality assessment. Vlado Menkovski, MSc Eindhoven University of Technology, NL Email: v.menkovski@tue."

Florence Anthony
8 years ago
Views:

1 Adaptive testing for video quality assessment Vlado Menkovski, MSc Eindhoven University of Technology, NL

2 Video characteristic diversity / V.Menkovski@tue.nl PAGE 1

3 Device diversity / V.Menkovski@tue.nl PAGE 2

4 Diversity in network services Internet Mobile data services Home Network Office Network / V.Menkovski@tue.nl PAGE 3

5 Diversity in content / electrical engineering PAGE 4

6 User aware Network management Perceived Quality / V.Menkovski@tue.nl PAGE 5

7 Rating Mean Opinion Score (MOS) / electrical engineering PAGE 6

8 DMOS variability based on data accessible at / electrical engineering PAGE 7

9 Rating labels vagueness MOS Quality Impairment 5 Excellent Imperceptible 4 Good Perceptible but not annoying 3 Fair Slightly annoying 2 Poor Annoying 1 Bad Very annoying Very Satisfied Satisfied Some Users Satisfied Many Users Dissatisfied Most Users Dissatisfied / electrical engineering PAGE 8

annoying 4.4-5.0 Very Satisfied 4.0-4.3 Satisfied 3039 3.0-3.9 Some Users Satisfied 2.

10 Improve Subjective testing 2 Alternative forced choice vs. Rating / V.Menkovski@tue.nl PAGE 9

11 Psychometric testing Psychophysics y quantitatively investigates the relationship between physical stimuli and the sensation of perception / V.Menkovski@tue.nl PAGE 10

12 Psychometric testing The horizontal axis of the Figure represents the physical intensity of the stimuli (amount of bit-rate in the video). The vertical axis is the difference scale, and represents the internal scale for perceived quality of the videos in relation to each other. / V.Menkovski@tue.nl PAGE 11

The vertical axis is the difference scale, and represents the internal scale

13 Maximum likelihood difference scaling Which of the two pairs has a bigger difference / V.Menkovski@tue.nl PAGE 12

14 MLDS / V.Menkovski@tue.nl PAGE 13

15 MLDS / electrical engineering PAGE 14

16 MLDS The difference between the first and the second pair is positive / electrical engineering PAGE 15

17 MLDS However, the response is contaminated with a Gaussian noise 0 - / electrical engineering PAGE 16

18 MLDS - 0 / electrical engineering PAGE 17

19 MLDS The probability of selecting the first pair - 0 / electrical engineering PAGE 18

20 MLDS The probability of selecting the second pair - 0 / electrical engineering PAGE 19

21 MLDS The likelihood for all the responses is the multiplication of all individual probabilities L(, ) ( ) 1 ( ) 1 ( ) 1 ( )... / electrical engineering PAGE 20

22 MLDS The likelihood of a set of test would be L(, ) ( ) 1 ( ) 1 ( ) 1 ( )... Given R 1 1 R 2 0 R 3 0 R ; This leaves es one equation with 10 unknown n parameters,,,..., / electrical engineering PAGE 21

23 MLDS Maximizing i i likelihood lih,,,..., Using a generalized linear model to estimate / V.Menkovski@tue.nl PAGE 22

24 Subjective Experiment 10 Different video with 10 different CBR bitrate settings: 2Mbps, 1.5Mbps, 1Mbps,, 256kbs, 128kbs, 64kbps / V.Menkovski@tue.nl PAGE 23

25 Results Relative difference Reference Video Perceived quality vs. bit-rate for 10 CBR videos / V.Menkovski@tue.nl PAGE 24

26 Results standard error / V.Menkovski@tue.nl PAGE 25

27 Fitting a psychometric curve Highly sensitive zone / V.Menkovski@tue.nl PAGE 26

28 Results - Psychometric curves Utility of perceived quality over bit-rate / V.Menkovski@tue.nl PAGE 27

29 Results / electrical engineering PAGE 28

30 Results / electrical engineering PAGE 29

31 Adaptive MLDS 10 For each video we need: Is this necessary? tests m k j i / electrical engineering PAGE 30

32 Adaptive MLDS 10 For each video we need: Is this necessary? tests m k j i k j i l f( x ) ; f( x ) ; f( x ) ; f( x ) ; f( x ) ; i i j j x x x x x i j k l m k k l l m m / electrical engineering PAGE 31

33 Adaptive MLDS k l k m f ( x ) ; f ( x ) ; f ( x ) ; f ( x ) ; f ( x ) ; i i j j k k xi xj xk xl xm i j k l m l l m m xk xl xk xm k l k m xk xl xk xm / electrical engineering PAGE 32

34 Adaptive MLDS If test t T1(x1, 1 x2, x3, x5) first pair is bigger then T2(x1, x2, x3, x4) first pair is bigger as well T1 > T2 > T3 T4 > > < T1 T5 < T6 < / electrical engineering PAGE 33

35 Adaptive MLDS How sure are we in the response of the user to T1? We can actually calculate it, if we know the Ψ values. - 0 / electrical engineering PAGE 34

36 Adaptive MLDS Combining i probabilities biliti PAB ( ) 0.7 PAC ( ) Assumed da and B are independent events and that the answers Y and N symmetric PABC (, ) PABPAC ( ) ( ) PABPAC ( ) ( ) (1 PAB ( ))(1 PAC ( )) PABC (, ) 0.77(7) / electrical engineering PAGE 35

37 Adaptive MLDS 1. Select a random batch of tests 2. Calculate the Psi values using MLDS from test results 3. Calculate the estimates for the remaining tests based on the dependencies and the probabilities of the answers 4. Select a number of tests that have the most uncertain estimates and get the responses for them 5. If the uncertainty in these tests is higher than the desired level el then go to step 2, otherwise finish. / electrical engineering PAGE 36

38 Results / electrical engineering PAGE 37

39 Results Number of results introduced: / electrical engineering PAGE 38

40 Results / electrical engineering PAGE 39

41 Thank you! / V.Menkovski@tue.nl PAGE 40

Analyzing Mission Critical Voice over IP Networks. Michael Todd Gardner

Analyzing Mission Critical Voice over IP Networks Michael Todd Gardner Organization What is Mission Critical Voice? Why Study Mission Critical Voice over IP? Approach to Analyze Mission Critical Voice