Personal TV Channels

Transcription

1 Examensarbete LITH-ITN-MT-EX--07/042--SE Personal TV Channels Per Carlsson Department of Science and Technology Linköpings universitet SE Norrköping, Sweden Institutionen för teknik och naturvetenskap Linköpings universitet Norrköping

2 LITH-ITN-MT-EX--07/042--SE Personal TV Channels Examensarbete utfört i medieteknik vid Linköpings Tekniska Högskola, Campus Norrköping Per Carlsson Handledare Reiner Lenz Handledare Ad Denissen Examinator Reiner Lenz Norrköping

3 Avdelning, Institution Division, Department Institutionen för teknik och naturvetenskap Datum Date Department of Science and Technology Språk Language x Svenska/Swedish Engelska/English Rapporttyp Report category Examensarbete B-uppsats C-uppsats x D-uppsats ISBN ISRN LITH-ITN-MT-EX--07/042--SE Serietitel och serienummer ISSN Title of series, numbering URL för elektronisk version Titel Title Personal TV Channels Författare Author Per Carlsson Sammanfattning Abstract The Personal TV Channel concept, still in an early stage of research, will record all TV programmes of interest in separate virtual channels (news channel, TV-series channels, movie channels) based on your preferences. The concept has been well received and this report investigates the technical issues and possibilities of creating Personal TV Channels. The development was done mostly as stand alone applications but also by extending to the open-source Linux PVR software MythTV. The experimental results using video fingerprint techniques shows a high accuracy for finding the starting points for TV-series and news broadcasts. The developed visualisation tool provides a clear output of the broadcast segmentation. The segmentation and matching of commercials outputs extensive commercial statistics and makes it possible to track the broadcast of specific commercials. Finally conclusions and future work are presented. Nyckelord Keyword Video Scene Boundary Detection, Commercial Detection,Video Fingerprinting, Video Indexing, Content Analysis, Pattern Recognition

4 Upphovsrätt Detta dokument hålls tillgängligt på Internet eller dess framtida ersättare under en längre tid från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: Per Carlsson

5 Final thesis Personal TV Channels by Per Carlsson

6

7 Final thesis Personal TV Channels by Per Carlsson Supervisor : Principal System Architect Ad Denissen Storage Systems and Applications at Philips Research Laboratories Examiner : Reiner Lenz Dept. of Computer and Information Science at Linköpings universitet

8

9 Abstract The Personal TV Channel concept, still in an early stage of research, will record all TV programmes of interest in separate virtual channels (news channel, TV-series channels, movie channels) based on your preferences. The concept has been well received and this report investigates the technical issues and possibilities of creating Personal TV Channels. The development was done mostly as stand alone applications but also by extending to the open-source Linux PVR software MythTV. The experimental results using video fingerprint techniques shows a high accuracy for finding the starting points for TV-series and news broadcasts. The developed visualisation tool provides a clear output of the broadcast segmentation. The segmentation and matching of commercials outputs extensive commercial statistics and makes it possible to track the broadcast of specific commercials. Finally conclusions and future work are presented. Keywords : Video Scene Boundary Detection, Commercial Detection, Video Fingerprinting, Video Indexing, Content Analysis, Pattern Recognition iii

10 iv

11 Acknowledgements Ad Denissen for supervising my internship with enthusiasm and knowledge. Ingmar Van Dijk and Ard Biesheuvel for your expertise in Linux, programming and software architecture. Adolf J. Proidl for your expertise in Personal TV Channels. Thank you. v

12 vi

13 Contents 1 Introduction Background Philips Research PVR Personal TV Channels Electronic Program Guide Main Goal Report structure Limitations Video input MythTV Commercial detection Method Abbreviations, acronyms and definitions TV Programme Boundaries Detection TV channel characteristics Broadcast structure Statistics Video Fingerprint Subtractive Fingerprint Fingerprint Generation Fingerprint Matching Matching issues vii

14 viii CONTENTS 2.5 System Architecture Visualisation XSLT Image Extraction Video Extraction Skiplists Commercial matching Commercial statistics Tracking of specific commercials Experimental Results The intro issue Preknowledge of the headers Conclusions Future work Appendix A 36 Bibliography 38

15 Chapter 1 Introduction 1.1 Background The era of waiting for TV programmes to begin is over. The user wants to watch his favourite shows when he turns on his TV. Instead of zapping through the channels waiting for something to come on just turn on your own channel and start watching the latest episodes of your preferred content. Personal TV Channels makes this possible. 1.2 Philips Research This thesis has been made for the Storage Systems and Applications group at Philips Research, Eindhoven. Founded in Eindhoven, The Netherlands, in 1914, Philips Research as part of Koninklijke Philips N.V. has expanded the scale and scope of its activities to become one of the world s major private research organisations. The annual research budget of Philips Research is slightly less than 1% of Philips annual sales, which amounted EUR 30.4 billion in Roughly two-thirds of the corporate research work is geared to the activities of the Product Divisions of Philips, with contractual agreements about programs and costs. The remainder is research of a more exploratory nature. (Total 1

16 PVR R&D efforts of Philips amount to approximately 8% of sales.) [1] 1.3 PVR A Personal Video Recorder (PVR) is a device used to watch and digitally record television broadcasts. Usually, a PC with a TV-tuner card is used as hardware base, but it is the software that defines the type of PVR. Different PVR software are for example ReplayTV, TiVo, Microsoft Media Center and MythTV. The PVR is meant to replace the regular TV-tuner, optical disc player and video recording device in the household. By having the video broadcast go through a PVR instead of directly to the television set, there are new possibilities to use the abilities of the PC which has much more processing power, memory and storage capabilities. On a PVR, the received video broadcast is first stored in a buffer in the memory of the PC before being displayed on the screen. This gives the ability to instantly record the material that is being watched, since this operation only requires the video stored in the buffer to be written to any storage device such as a hard drive. This also allows time shifting where the user can press pause in a live programme. The PVR will immediately start recording the remains of the programme and the user can then continue to watch the recording from that point at another time. For Personal TV Channels its even easier. Limitless fast forward and backward is enabled without the need of any buffer since all the data is stored on the hard drive. 1.4 Personal TV Channels Personal TV Channels also known as Flex channels is under research by Philips, [2] and [3]. Personal TV Channels lets the user create TV channels containing only programmes chosen by the user. Examples are recordings of all news broadcasts from a certain channel or every movie starring Arnold Schwarzenegger. All programmes recorded for the Personal TV Channel will be played in a specific order when the user selects the channel, creating the illusion of a regular television channel. For this feature, precise boundaries detection will be important since the need to fast forward in any

17 Introduction 3 direction ruins the illusion of a regular TV channel. Personal TV Channels are made with the remote of your PVR and watched like any other channel on your TV, see image 1.1. The concept of Personal TV Channels is a future differentiator separating Philips PVR boxes from other PVR boxes. Figure 1.1: Regular and Personal TV Channels in seamless integration 1.5 Electronic Program Guide The Electronic Program Guide (EPG) is extra information about the broadcast often send as side information in a digital video broadcast, but can also be retrieved in other ways (for example by downloading from a remote server). The EPG can contain some information about the programmes, for example start and end time, starring actors, a short description etc. The schedule information is very useful for scheduling recordings in advance. But the EPG does not contain exact time information. If the exact starting and end time of each programme was available, it would be very easy to find the start and end points, and the channel would not get any advertisement money. The times are not exact because the commercial breaks are not shown in the EPG. As a user, however, we are usually interested in the material we recorded, and not additional surrounding advertisement. Therefore, enhancement of the information in the EPG is wanted to make it more accurate to fit our needs.

18 Main Goal 1.6 Main Goal The main goal is automatic programme boundaries detection using content analysis and pattern recognition of recorded TV programmes to make Personal TV Channels. 1.7 Report structure First some preliminaries and definitions are given. Then the method and the work process are described followed by experimental results, conclusions and future work. 1.8 Limitations The method is limited to the pixel domain (no use of the compressed domain). No limitation to real-time performance. The method is focused on TV-series and news broadcasts running on some of the most popular television channels in the Netherlands. The method is supposed to work on full recorded programmes that contains a few minutes of additional recorded material before the beginning and after the end of the programme. For the segmentation and matching of commercials TV channels are expected to include blank frame segments between commercials. 1.9 Video input The video is recorded as MPEG-2 PS with a resolution of 720*576 at 25 frames per second with a bit rate of 4500 kbps (roughly 2 GB/hour) using a MPEG-2 hardware encoder and the MythTV software MythTV MythTV is a popular open-source PVR software program for Linux under the GNU license. For this project version 0.20 were used. See image 1.2.

19 Introduction 5 Figure 1.2: Snapshot of the Linux PVR software MythTV showing the EPG information based recording interface Commercial detection Commercial detection algorithms are typically based on the detection of blank frames, channel logotypes and scene change frequency. Commercial detectors ([4], [5], [6], [7], [8]) are able to detect most of the commercials in a recording but are still unable to find the starting points of programmes because of two reasons. The first one is that many programmes start right after the end of the previous one. The second one is that there are other content except the commercials and the main programme in the recording like trailers for upcoming shows and parts of other programmes Method Instead of trying to cut out all commercials and content that is not the main programme we are explicitly looking for the start of the programme. The proposed method uses mainly video fingerprinting but also some commercial detection techniques are used. All the content is left intact on the

20 Abbreviations, acronyms and definitions hard disk of the PVR, the output of the method is only a pointer to the starting point of the main programme in the recording. See figure 1.3. Figure 1.3: Recording consisting of main programme and other content such as other programmes and commercials 1.13 Abbreviations, acronyms and definitions PVR Personal Video Recorder PC Personal computer MPEG Motion Pictures Experts Group MPEG-x Standard number x developed by MPEG TV Television Set-top box A set-top box describes a device that connects to a television and some external source of signal, and turns the signal into content then displayed on the screen. Ground Truth The manually annotated truth used to help determine the accuracy of the automatic classification by comparison. EPG Electronic Programme Guide MythTV Open-Source PVR software

21 Chapter 2 TV Programme Boundaries Detection In the segmentation we focus on finding the starting point of the main programme in the recording. Finding the starting point is more important than finding the ending point because as long as you can easily skip forward to the precise starting point of the next show in the sequence of your Personal TV Channel it s not vital that the shows are completely seamlessly composed. 2.1 TV channel characteristics TV channels can be divided into three types based on the commercials. TV channels without commercials, TV channels with commercials only in between programmes and TV channels with commercials anytime. Finding the starting point will be easier if you always have commercials in between and only in between programmes. Finding the starting point will be more difficult without commercial or with commercial anytime. Our method aims at handling all these types of TV channels. 7

22 Broadcast structure 2.2 Broadcast structure An extensive study of the different segments of the broadcasting structure and a few terms was introduced for easier description of a recording. Notice how intro and header is defined since these words will be used frequently and the use of these terms is not evident. Main Programme The main programme of the recording. Other Programme Another programme in the recording. Commercial A single commercial. Usually lasts 4-40 seconds. Self Commercial Self Commercials are trailers for upcoming shows in the near future. Usually lasts 5-40 seconds. Also referred to as selfcoms. Blank frame segment Four to eight consequent blank (black) frames separating commercials. Commercial Separators Commercial separators, usually 3-15 seconds, informs the viewers about the start or end of a commercial block. Also referred to as CommSeps. See image 2.1. Sponsor The programme is sponsored by a company. See image 2.2. Intro Introduction part of the programme before the header, not present in all programmes. Header When a programme starts it usually starts with a header, an introduction signature sequence that is always the same in all programmes. See image 2.3. Government Commercial Commercial from the government are different in that they do not follow the usual regulations for commercials with blank frames and commercial separators. Figure 7.1 (in the appendix) shows the first part of two manually annotated episodes of Goede tijden, slechte tijden. Each segment is displayed with an representative image in combination with key numbers and text.

23 TV Programme Boundaries Detection Statistics Figure 2.1: Commercial Separator The manually annotated episodes of Goede tijden, slechte tijden shows some interesting statistics, see table 2.1 and 2.4. Most striking is that when popular shows (popular shows contain more commercials) are recorded with some extra minutes around the recording because of an inaccurate EPG the percentage of main programme in the recording is low, close to 50%. This is expensive in storage space and clearly indicates the need to segment the recording in order to be able to make Personal TV Channels. 2.4 Video Fingerprint To find the starting points of the video recordings we are using video fingerprints. By using video fingerprints we are able to detect repeating parts of a TV channel. Parts that are repeating are supposed to belong to either a ComSep, a SelfCom, a header or a commercial. A video fingerprint identifies a video sequence like a human fingerprint identifies a human. Like human fingerprint they should be robust against distortions and transformations. More specific it should have robustness to

24 Video Fingerprint Figure 2.2: Sponsor segment subtle differences in rotation, scaling, luminance and colour. And it should be easy and fast to calculate and include only a small number of values so that storing and comparing calculations are inexpensive. The fingerprint method selected is Differential Variable Size Block Luminance Algorithm, [9]. A method based on mean luminance values extracted from a grid as seen in figure 2.5. It is robust to rotation and scaling, quick to generate and simple enough for fast matching calculations. The fingerprint is small using only 38 numbers per frame namely the mean value, the variance and 36 area luminance numbers Subtractive Fingerprint The selected fingerprint method recommends storing the differences between block i and block i+1 instead of just the luminance value. A decision was made not to do this since we are also using the fingerprint to find the black frames between commercials. But storing the luminance and not the differences has some issues. If a matching sequence is overall slightly brighter the match will be negative even though the difference values would match perfectly. The solution would be to store the differences between block i and block i+1 to improve the matching but to store the mean and variance based on

25 TV Programme Boundaries Detection 11 Figure 2.3: Header of the popular TV-series Goede tijden, slechte tijden the original luminance values to still be able to detect black frames using the fingerprint Fingerprint Generation To generate fingerprints from a large number of recordings fingerprint generation were added to MythTV. A large of number of TV-series and news broadcasts from some of the most watched Dutch TV channels were recorded and fingerprinted Fingerprint Matching The matching intensity is calculated per frame and compared to a threshold to decide if the match is good enough. Then the length of each matching sequence is counted, see table 2.2. All but the longest match are ignored. Each matching sequence has a time offset between the two compared files. This offset, in frames, occurs where the diagonal line from the match intersects the axes. In table 2.2 the longest matching sequence is matching for seven letters with an offset of one for the vertical episode and three for the horizontal episode. The match of two fingerprints can easily be displayed as an image with

26 Video Fingerprint Figure 2.4: Commercial length histogram of six episodes of Goede tijden, slechte tijden Table 2.1: Statistics of two manually annotated episodes of Goede tijden, slechte tijden, seconds in parenthesis 30 number of commercials per recording 524 (20.24) number of frames per commercial in average 124 (4.24) and 1124 (44.24) min and max length of commercial 26.7% percentage of recording that is commercial 54.6% percentage of recording that is main programme 18.7% percentage of recording that is not commercial or main programme 147 (5.22) number of frames per commercial separator in average 73 (2.23) and 424 (16.24) min and max length of ComSeps 7 number of blank frames in between commercials in average 6 number of blank frames in between commercials and ComSeps in average

27 TV Programme Boundaries Detection 13 Figure 2.5: Fingerprint consisting of mean luminance values matching similarity as intensity. See figure 2.6 and 2.7. Blank frames shows up as squares. Matching headers/commercials as lines Matching issues The counting algorithm has been tweaked to be more forgiving when handling two types of matching issues. One is the issue of missing frames as seen in figure 2.8 where segments are fully matching except for a missing frame in one of the segments breaking the perfectly straight line. The other one is the issue of partial weak matches as seen in figure 2.9 where the segments are matching completely except for a few frames. 2.5 System Architecture In the most simple case MythTV generates the fingerprints. Then a stand alone application inputs the fingerprints, does the segmentation and matching and outputs a text file of the results. See figure 2.10

28 System Architecture Y B B B A S P I L I H P R B B B P H I L I P S R E S Table 2.2: Matching concept Figure 2.6: Match image

29 TV Programme Boundaries Detection 15 Figure 2.7: Match between parts of two recordings, the two white lines are matching segments, the rest of the white parts mostly blank frames matching Figure 2.8: Match image - missing frame

30 System Architecture Figure 2.9: Match image - weak match Figure 2.10: Initial System

31 Chapter 3 Visualisation Figure 3.1: Complete system architecture To make the segmentation and matching output clear, there is a need for a meaningful visualisation. To accomplish this the system was extended, 17

32 XSLT see figure 3.1. An XML file is output and to close the loop, skip-lists are output and executed to update the MythTV database with the starting points found in the stand alone application. Also output are images and video sequences extracted from the recordings. 3.1 XSLT Extensible Stylesheet Language Transformations (XSLT) is an XML-based language used for the transformation of XML documents. In this case XML to HTML using Mozilla Firefox as the XSLT processor. The main system output is an XML file containing the complete segmentation data. The XML file is then transformed to a HTML representation using a XSLT code. Different style-sheets transform the same XML in seconds using a standard web browser to different output representations. Maybe you want a simple text list of the length of found commercials for further external plotting or maybe you want a more complete visual output with representative images, see figure Image Extraction For each segment a representative image is extracted. An image is chosen according to time and matching width (normally the brand name is displayed for a few seconds in the end of the commercial). This is most likely a good snapshot of the commercial for visualisation. Figure 3.3 shows with a dotted line where the image was extracted for a specific match. 3.3 Video Extraction The application is also capable of outputting Flash Video files as an alternative to the representative images. This gives a more complete representation with the possibility to forward/backward but it s considerably more expensive in extraction time and storage space. Flash Video (.flv files) using H.263 for encoding the video and MP3 for encoding the audio was used.

33 Visualisation Skiplists It is possible to set cut-lists for recordings using MythTV. A cut-list is of the form: #-#[,#-#]... (ie, 1-100, , ) and stored in the MythTV database. The stand alone application outputs and executes skip-lists to update the recordings with the found starting point which is then used when recordings are played in MythTV.

34 Skiplists Figure 3.2: Visualisation with segments coloured according to type and also according to matching results

35 Visualisation 21 Figure 3.3: Representative image extraction

36 Chapter 4 Commercial matching Commercial detection methods are advanced and a well explored field, ([4], [5], [6], [7], [8]). For this project only a simple commercial detection algorithm was implemented since the main focus was on video fingerprint methods. Blank frame detection was chosen since it s the foundation of most modern commercial detectors and since it does not only find the commercial block but the starting and end point of most of the individual commercial within the commercial block. In the Netherlands there is a law that television broadcasters must separate commercials with blank frames. Commercials are found in most cases. Problems occur when there are sometimes no blank frames between the commercial separators and the commercials (the first and last commercial of the commercial block not always found) and when there are long black fades in a commercial (one commercial detected as several commercials). Figure 5.4 shows the difference between the last automatically found commercial and the actual last, manually annotated, commercial. Figure 4.1 gives an rough idea of how many commercials are matching when a large number of recordings are matched. 63 commercials were matched and a subset of them are displayed. Coloured fields represents matching commercials. 22

37 Commercial matching 23 Figure 4.1: Commercial matching for a large number of recorded programs 4.1 Commercial statistics The automatic finding of a large number of commercials gives the opportunity to output extensive commercial statistics like the one seen in table 2.1 but for a large number of episodes. But the statistics are not perfectly accurate because of the minor occurring problems in the automatic blank frame based commercial segmentation. 4.2 Tracking of specific commercials Each commercial is assigned a unique id. When visualising the matching data this id number can be used to highlight only specific commercials using a condition statement. See figure 4.2.

38 Tracking of specific commercials Figure 4.2: Highlighting of two specific commercials in three recordings of Goede tijden, slechte tijden

39 Chapter 5 Experimental Results The performance of the developed algorithm for finding the programme starting points was tested on a number of recordings by measuring the distance in time between the actual ground truth starting point and the starting point found by the algorithm. Table 5.1 and 5.2 displays the offset between the found starting point and the actual starting point for an increasing number of input files of different kinds of recordings. For the Teletubbies episodes of table 5.1 the error is decreasing for an increasing number of episodes. For the Bold and the Beautiful episodes of table 5.2 the result is about ten seconds off since the ComSep before the episode is identical for all tested episodes. The result will also be off if the programme before has an identical tail or if there are identical SelfComs. But with the increase of recordings of the same type, the surrounding material that is not wanted will hopefully cancel out as noise as more recordings of the same type will give a more outstanding hit for the wanted repeating programme header. Table 5.3 shows that matching of only two episodes is generally not enough to obtain an accurate starting position. Finding the starting points of a programme can be done faster since I use commercial detection to get an idea of the recording and only do matching where the occurrence of a header is likely, see figure 5.1. The two main types of TV programmes I was looking at was TV-series 25

40 26 e1 e1,e2 e1,e2,e3 e1,e2,e3,e4 e1,e2,e3,e4,e5 e2 01:19:06 e3 00:00:13 00:00:06 e4 00:11:06 00:11:06 00:00:19 e5 00:01:16 00:01:16 00:02:05 00:00:19 e6 00:01:10 00:00:06 00:00:29 00:00:19 00:00:19 Table 5.1: Time offsets for episode 1 of Teletubbies when matched with an increasing number of other episodes e1 e1,e2 e1,e2,e3 e1,e2,e3,e4 e1,e2,e3,e4,e5 e2 00:10:21 e3 00:07:03 00:10:10 e4 00:09:01 00:09:01 00:09:01 e5 00:19:13 00:09:04 00:10:06 00:10:06 e6 00:19:15 00:09:06 00:10:06 00:09:11 00:10:06 Table 5.2: Time offsets for episode 1 of Bold and the Beautiful when matched with an increasing number of other episodes Teletubbies Bold and the Beautiful episode gt offset error gt offset error 2 06:40:12 07:59:18 01:19:06 05:57:12 05:46:16 00:10: :40:12 06:41:00 00:00:13 05:57:12 06:04:12 00:07: :40:12 06:29:06 00:11:06 05:57:12 05:48:11 00:09: :40:12 06:42:03 00:01:16 05:57:12 06:17:00 00:19: :40:12 06:41:22 00:01:10 05:57:12 06:17:02 00:19:15 Table 5.3: Time offset for episode 1 when matched with five similar episodes

41 Experimental Results 27 Figure 5.1: Sections likely to contain the header and news broadcasts. The differences between TV-series and news broadcasts are displayed in table The intro issue Figure 5.2 shows the offset from the actual starting point of the header to the automatically found header for 50 recordings, it does not take into account the actual starting points of the programs with an intro. In other words, intros are not considered. Given the header and the last commercial you would think it is easy to find the beginning of the episode. But there are some significant difficulties. First the intro length varies as seen in fig-

42 Preknowledge of the headers Tv-serie News broadcast header length long short header automatically found yes no intro sometimes sometimes Table 5.4: Comparison between TV-series and news broadcasts ure 5.3. Moreover, as seen in figure 5.4, there is also a difference between the last automatically found commercial and the actual last, manually annotated, commercial. Also commercial is lacking in some cases just before the programme. Figure 5.4 also illustrates how much the structure varies even for the same TV-series on the same TV-channel. A basic solution would be to go back two minutes from the beginning of the header unless the last commercial are found closer then these two minutes, in that case go back to the end of the last found commercial. 5.2 Preknowledge of the headers For news broadcasts the starting points can be found accurately if one header is given. One way of providing this knowledge would be to set up a server that holds the fingerprint of the start sequence for every news show. It could also be possible to allow the users to do it. I.e. as soon as one user takes the effort to tell one box where the start/stop of the intro is the fingerprint information is shared back with the server and made available to everybody.

43 Experimental Results 29 Figure 5.2: Offset between the found starting point of the header and the actual starting point of the header in seconds for 50 recordings

44 Preknowledge of the headers Figure 5.3: Length of TV-series ONM intro in seconds for 14 episodes

45 Experimental Results 31 Figure 5.4: Intros for nine episodes of ONM

46 Chapter 6 Conclusions Finding the programme starting points using video fingerprint techniques was the main task for me in this project. The experimental results using my developed algorithm show a high accuracy for finding the starting points. Starting points in TV-series can be found without knowing the header. But TV-series sometimes have intros and going from the header to the beginning of the programme is not trivial, see discussion in 5.1. Starting points in news can be found automatically but only if the header is given for one of the episodes. This is a big drawback, see discussion in 5.2. While working with the project I was frustrated trying to read the text output generated by the matching and I decided to develop a XML to HTML visualisation tool to enhance the presentation. The developed visualisation tool provides a clear output of the broadcast segmentation using not only text but tables, colors and images. I was asked during the project to look into the topic of target advertising for possible future implementation as a part of Personal TV Channels. The segmentation and matching of commercials I developed outputs extensive commercial statistics and makes it possible to track the broadcast of specific commercials using the convenient XSLT processing language. My results contribute to the current Personal TV Channel knowledge but more research is needed before a full Personal TV Channel demo can be developed. 32

47 Conclusions Future work There are numerous possibilities for future work ranging from a very practical to a more conceptual level. The segmentation and matching of commercials shows interesting future possibilities for target advertising. Commercials in recordings could be exchanged with up to date commercials personalised for the user. Using the video fingerprint as intended storing luminance differences instead of just luminance values would improve the matching, see discussion in section The brute force method for matching the fingerprints could be approved by using more advanced methods proposed in [9] to significantly increase the matching speed. There are some unresolved issues with the memory management in the main stand alone matching / visualisation program for matching of large segments, see figure 6.1. The procedures of allocation of memory at request and freeing it for reuse when no longer needed could be improved. The database connection to the stand alone application is not fully developed. Building a database with commercials and headers would eliminate time consuming and redundant rematching, see figure 6.2. Other solutions then ignoring all but the longest match could be evaluated when multiple matches are found such as choosing the one closest to the EPG starting time if you are looking for a header.

48 Future work Figure 6.1: Memory management issue

49 Conclusions 35 Figure 6.2: Database added to the system architecture

50 Chapter 7 Appendix A 36

51 Appendix A 37 Figure 7.1: Ground truth segmentation of 2 episodes of Goede tijden, slechte tijden

52 Bibliography [1] [2] A.J. Proidl et al M. Verberkt. Virtual channels. Technical report, Philips Restricted, [3] S.P.P. Pronk A.J. Proidl. Intelligent virtual channels. Technical report, Philips Restricted, [4] B. Satterwhite and O. Marques. Automatic detection of tv commercials. In IEEE Potentials, [5] A Albiol A Albiol, M.J.C. Fullà and L Torres. Detection of tv commercials. In IEEE International Conference on Acoustics, Speech and Signal Processing, [6] L. Agnihotri and N. Dimitrova. Evolvable visual commercial detector. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, [7] Dieter W Blum. Method and apparatus for identifying and eliminating specific material from video signals. [8] W. Effelsberg R. Lienhart, C. Kuhmunch. On the detection and recognition of television commercials. In IEEE International Conference on Multimedia Computing and Systems, [9] V. Bhargava. Video fingerprinting. Technical report, Philips Restricted,

53 Copyright Svenska Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida English The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: c Per Carlsson Linköping, September 1, 2007