Novi Sad Breakout: Joint Qualinet Crowdsourcing Experiment

Size: px

Start display at page:

Download "Novi Sad Breakout: Joint Qualinet Crowdsourcing Experiment"

Shonda Young
8 years ago
Views:

1 Novi Sad Breakout: Joint Qualinet Crowdsourcing Experiment 30 Minutes Session 1. Collect information 2min Your Ideas? Your Contribution? 20min 2. Discussion 10min t 1

2 Your Ideas! What is your interest in the crowdsourcing experiment? Develop and apply methodology (e.g. model validation, e.g. merging two subjective user studies) Derive a new QoE model (e.g. for gaming QoE) Analyze impact of crowdsourcing environment (e.g. due to incentives) Which concrete research questions do you want to answer? Is there an impact of demographics on video quality assessment? Hypothesis: Machine learning models based on crowdsourcing results will lead to higher uncertainty than lab tests. Which application to investigate? DASH and SVC VoIP Web Browsing (serious) gaming Image quality Radio streaming 2

Is there an impact of demographics on video quality assessment?

3 Which application? Which selection criteria? Application Lab results available / contributor #Crowdsour cing issues: bandwidth, reliability, #Research questions addressed #VIPs (very interested people = contributor)??? 2D video streaming Adaptive streaming VoIP Web browsing Cloud storage Gaming Images Radio streaming 3

bandwidth, reliability, #Research questions addressed #VIPs (very interested

4 Your Contribution! Who is willing to contribute what? Task Laboratory test Crowdsourcing test Design of user test Realistic test parameters Implementation of test Execution of test Data analysis Research Question 1 Research Question 2 How to implement the test? Which concrete platform to use? How should concrete tests look like? How to design a nice game? 4

Implementation of test Execution of test Data analysis Research Question 1 Research

5 Next Steps 1. Summary via mailing list / wiki Your interests Your contributions 2. Collective decision within TF Collect info from all TF participants Google survey form 3. Online meeting Decision on concrete application, platform, research questions Allocation of work for VIPs Rough time schedule 4. Time plan 15/03/2013: summary 22/03/2013: google survey sent around 31/03/2013: TF fills survey Mid april: online meeting 5

Online meeting Decision on concrete application, platform, research questions Allocation of work for VIPs

6 Input collected before Novi Sad meeting 6

7 Joint Qualinet Experiment What is the impact of the crowdsourcing environment on QoE results? What is the difference to lab tests? Regarding test duration, as only a subset of test conditions can be tested in crowdsourcing. Regarding demographics, as users are world-wide, may have different expectations, usage habits, etc Regarding incentives, e.g. a test implemented as game vs. a pure test What are statistical methods to analyze data? Can we merge data from lab and crowdsourcing? Reliability of users and consistency of user ratings Quantification of data quality 7

Regarding demographics, as users are world-wide, may have different expectations, usage habits, etc Regarding incentives, e.g. a test implemented as game vs.

8 Concrete Steps for Joint Qualinet Experiment Which application to investigate? Existing models and subjective user ratings for comparison, e.g. SVC Developing new QoE models, e.g. HDR images Maybe image quality to simplify the implementation due to Internet delivery and bandwidth bottlenecks as for video Interest by other Qualinet Task Forces? Which concrete research questions to tackle? Impact of crowdsourcing environment: Do reliability questions impact the user ratings? Like counting quality switches? Incentive design: design QoE test as game to improve quality/avoid cheating and get more users for free How to implement the test? Which concrete platform to use? How should concrete tests look like? How to design a nice game? Who is doing what? Implementation of test, execution of test, statistical analysis of results, etc. 8

Which concrete research questions to tackle? Impact of crowdsourcing environment: Do reliability questions impact the user ratings? Like counting quality switches?

9 Interest in Joint Qualinet Experiment Filippo Mazza, Patrick le Callet, Marcus Barkowsky: comparison of lab and crowdsourcing experiments considering model validation; directly related to Validation TF Martin Varela, Lea Skorin-Kapov: impact of crowdsourcing environment on user results and QoE models, e.g. incentives and payments on the example of Web QoE; directly related to Web/Cloud TF Christian Keimel: Impact of crowdsourcing environment on user results and QoE models, e.g. demographics Andreas Sackl, Michael Seufert: Impact of content/consistency questions on QoE ratings, e.g. for HTTP video streaming; directly related to Web/Cloud TF Bruno Gardlo: currently working on improved crowdsourcing platform with screen quality measurement etc.; interest in incentive design, gamification; platform may be used for experiment, e.g. for videos or images Katrien de Moor: contribution in the questionnaire development/refinement and/or by setting up a comparative lab test Babak Naderi: development of crowdsourcing micro-task platform which may be used for joint experiment; incentives, data quality control, effects of platformdependent and user-dependent factors on motivation and data quality 9

environment on user results and QoE models, e.g.

10 Summary from Breakout Session will be compiled soon! 10

Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing

1 Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing Tobias Hoßfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian Habigt, Klaus Diepold, Phuoc Tran-Gia Abstract Quality of