End-to-endDelayofVideoconferencingoverPacket

End-to-endDelayofVideoconferencingoverPacket POBox218,YorktownHeights,NewYork1598 T.J.WatsonResearchCenter-IBM MarioBaldiandYoramOfek SwitchedNetworks calltointeractnaturally,theend-to-enddelayshouldbebelowhumanperception-about withtheobjectiveofunderstandinghowitcanbecontrolledandreduced. lossprobability,andboundeddelay.inorderfortheparticipantsinavideoconference 1ms.Weidentifythecomponentsoftheend-to-enddelayinvariouscongurations Therstcontributiontotheend-to-enddelaycomesfromtheprocessingperformed Videoconferencingapplicationsrequirefromthenetworkguaranteedbandwidth,small Abstract onpicturesbeforesendingandafterreceivingthem;wecallthisprocessingdelay.the secondcontributionisthenetworkdelay.sincepicturesmustbedisplayedatthesame paceatwhichtheyhadbeencaptured,anyvariationintheprocessingandnetwork switchingwithtimedrivenpriority,and(iv)asynchronouspacketswitching.thestudy results,forvideoconferencingwithmpeg,independentofthenetworkload. delaymustbecompensatedbeforepicturesaredisplayed.thiscompensationisdoneby MPEGvideo(VBRandCBR)over:(i)dedicatedlinks,(ii)circuitswitching,(iii)packet showsthatacommontimereferenceusedwithtimedrivenprioritycanprovideadequate throughanumberofcongurations.westudythetransmissionofbothrawvideoand addingresynchronizationdelay,whichisthethirdcomponentoftheend-to-enddelay. Wedevisethesebounds,andhencetherelatedend-to-enddelay,goingstep-by-step 564767,fax+3911564799,e-mailbaldi@athena.polito.it VistitingstudentfromDipartimentodiAutomaticaeInformatica,PolitecnicodiTorino,phone+3911 1

2TheModel Contents 1Introduction 1.1ScopeoftheWork:::::::::::::::::::::::::::::::::5 2.2RequirementsforVisualQuality::::::::::::::::::::::::::8 2.1RequirementsforInteraction::::::::::::::::::::::::::::7 2.2.1BueringIssues::::::::::::::::::::::::::::::9 65 3TransmissionofRawVideo 2.3TheSystem:::::::::::::::::::::::::::::::::::::1 3.1DedicatedLinkBetweenSenderandReceiver::::::::::::::::::12 3.3PacketSwitchingwithTimeDrivenPriority:::::::::::::::::::16 3.2CircuitSwitching::::::::::::::::::::::::::::::::::15 3.3.1BasicPrinciplesofTimeDrivenPriority:::::::::::::::::16 3.3.2TransmissionofVideoandScheduling::::::::::::::::::18 3.3.3MultipleVideoconferenceCallsontheSameNetwork::::::::::22 11 4TransmissionofVBRMPEGVideo 4.1MPEGOverview::::::::::::::::::::::::::::::::::29 3.4AsynchronousPacketSwitching::::::::::::::::::::::::::23 3.4.1NetworkResynchronizationDelay:::::::::::::::::::::23 3.4.2TracShapingatNetworkBoundaries::::::::::::::::::26 4.3PacketSwitchingwithTimeDrivenPriority:::::::::::::::::::38 4.2DedicatedLinkBetweenSenderandReceiver::::::::::::::::::37 4.1.3BitGenerationRateandQuantization::::::::::::::::::3 4.1.2PredictiveCoding::::::::::::::::::::::::::::::29 4.1.1Intra-frameCoding:::::::::::::::::::::::::::::29 4.4AsynchronousPacketSwitching::::::::::::::::::::::::::54 4.4.1TracShapingatNetworkBoundaries::::::::::::::::::57 4.3.3SynchronizationbetweenEncoderandNetwork:::::::::::::46 4.3.4ComplexScheduling::::::::::::::::::::::::::::51 4.3.1ChoosingaBoundonPictureDimension::::::::::::::::4 4.4.2AdaptingtheEncodedVideoStreamtotheNetwork::::::::::58 4.3.5ReducingDecodingTime::::::::::::::::::::::::::53 4.3.2ControllingDimensionofEncodedPictures::::::::::::::::41 4.5CircuitSwitching::::::::::::::::::::::::::::::::::59 2

5TransmissionofCBRMPEGVideo 5.1Intra-frameCodingOnly::::::::::::::::::::::::::::::61 5.1.1CodingShapingDelay:::::::::::::::::::::::::::61 5.1.2RateControlFunction:::::::::::::::::::::::::::62 5.1.3DimensionofEncodedPictures::::::::::::::::::::::64 5.1.4StartupShapingDelay:::::::::::::::::::::::::::66 5.1.5ShapingDelayImplementation::::::::::::::::::::::68 6 5.3ExperimentalData:::::::::::::::::::::::::::::::::81 5.2Intra-frameandPredictiveCoding::::::::::::::::::::::::74 5.2.2CodingShapingDelayatConstantTargetRate:::::::::::::76 5.1.6ExperimentalData:::::::::::::::::::::::::::::69 5.2.1BueringIssuesRelatedtotheCodingShapingDelay::::::::::75 5.6PacketSwitchingwithTimeDrivenPriority:::::::::::::::::::95 5.5CircuitSwitching::::::::::::::::::::::::::::::::::95 5.4DedicatedLinkbetweenSenderandReceiver:::::::::::::::::::91 5.2.3CodingShapingDelayatConstantVisualQuality::::::::::::78 5.2.4StartupShapingDelay:::::::::::::::::::::::::::8 5.7AsynchronousPacketSwitching::::::::::::::::::::::::::11 5.6.1NetworkShapingDelay:::::::::::::::::::::::::::96 5.6.3End-to-endDelay::::::::::::::::::::::::::::::99 5.6.2ExcessResynchronizationDelay::::::::::::::::::::::97 AListofAchronymsandSymbols 6Summary 5.7.2Multi-HopConguration::::::::::::::::::::::::::15 5.7.1Single-HopPacketNetwork::::::::::::::::::::::::11 BTheMPEGEncodingStandard 5.7.3PacketizationandStartupShapingDelay:::::::::::::::::15 B.2Intra-codedPictures::::::::::::::::::::::::::::::::112 B.1GeneralPrinciples::::::::::::::::::::::::::::::::::111 B.2.1DiscreteCosineTransformation::::::::::::::::::::::112 18 111 17 B.3Predictive-codedPictures::::::::::::::::::::::::::::::116 B.2.4ControllingI-frameDimension:::::::::::::::::::::::115 B.2.3EntropyEncoding::::::::::::::::::::::::::::::114 B.2.2Quantization::::::::::::::::::::::::::::::::113 B.3.3DCT:::::::::::::::::::::::::::::::::::::117 B.3.1MotionEstimation:::::::::::::::::::::::::::::116 B.3.2MotionCompensation:::::::::::::::::::::::::::117 3

B.5PacketizationandStreaming::::::::::::::::::::::::::::119 B.4SignaltoNoiseRatio::::::::::::::::::::::::::::::::119 B.3.6ControllingP-frameDimension::::::::::::::::::::::118 B.3.5EntropyEncoding::::::::::::::::::::::::::::::118 B.3.4Quantization::::::::::::::::::::::::::::::::118 4

1Introduction TheMovingPictureExpertGroup(MPEG)encodingwasdesignedforstorageandone-way mainobjectivehasbeentoreducethecommunicationresourcesrequirementswhilemaintaininghighvideoquality. (playback)transmissionofqualityvideowhereend-to-enddelayisnotamajorconcern.the (i)forhearingtheend-to-end(sendertoreceiver)delayshouldbebelow1ms, (ii)thevideostreamshouldbesynchronizedwiththeaudiostream(a.k.a.lip-synchronization), videocommunications: avoidvariouspitfalls,asshowninthisreport. qualityofservice(qos)parametershasnotbeendone.suchastudyisneededinorderto UsingMPEGforvideoconferencingisnotnew.However,asystematicstudyofvarious (iii)themedia(audioandvideo)replayatthereceivingsideshouldbecontinuous(which impliesthatthereplaybuerisneveroverowedorunderowed). whichimpliesthatthevideoend-to-enddelayshouldbekeptbelow1msaswell,and Wedenethefollowingrequirementsforinteractive("face-to-face")real-timeaudioand thatthepersonontheothersideisnexttous.ifwehadveryhighquality3d(dimension) nexttous-thisisalsoknownastele-presenceorvirtualreality.sotheresultspresentedin video,thenwithoureyesopenwecouldthinkthatthepersonontheothersideisactually thisworkcanhavefurtherconsequencesonthedesignofvirtualrealitysystems. Themainconsequenceoftheaboverequirementsisthatwecan"closeoureyes"andthink qualityofeachpicture.weconcentrateontheminimizationoftheend-to-endprocessing eachofthemthusdecreasingtheend-to-enddelay.whilereducingtheend-to-enddelayit isimportanttomaintainthevideoqualitybyavoidingpacketlossandbymaintainingthe howitisinuencedbythetransmissionandnetworkqueueingdelay. delay,i.e.,weconsiderthedelayintroducedbythecombinationofencoding/decodingand Weidentifythecomponentsofthedelayandstudytheirnaturewiththegoalofreducing investigationintoseveralsteps.ineachstepweincreasethecomplexityofthesystemwe 1.1ScopeoftheWork Inordertohaveanin-depthunderstandingofthevideoconferencingproblemwedivideour analyze.bydoingsowecanisolateandunderstandthevariousdelaycomponents. types: 1.Dedicatedlink,whichallowsthevideostreamtousetheentirecapacityofthelink, Weconsidertwovariables:thenetworktypeandthevideotype.Therearefournetwork thanaveragevideostreamrate. possiblyinaburstymanner.thus,inthiscasethelinkcapacitycanbemuchhigher 5

2.Circuitswitching,whichallocatesafractionofthecapacitytoeachvideostream.In 3.Timedrivenpriority[15],whichgivesthereal-timetracperiodicpriorityoverother thiscasetheassumptionisthatthecircuitcapacityequalstheaveragevideostream rate. 4.Asynchronouspacketswitching,whichdoesnothavecommontimereferenceandeven ifpriorityisusedcongestioncanoccur.asaresult,thedelayjittercanbelargeand trac(bothreal-timeandnonreal-time).sincealltheswitchingelementsareusing packetlosscanoccur. timeornonreal-time)trac,i.e.,thereal-timetracisprotectedbya\re-wall",and thedelayjittercanbeboundedbyaconstant. network.asaresult,thereal-timetraccannotbeadverselyaectedbyother(real- acommontimereference,timedrivenpriorityfacilitatespipelineforwardinginsidethe eectofthevideoprocessingontheend-to-enddelay. 2.Variablebitrate(VBR)MPEG.Eachpictureisencodedandimmediatelysenttothe 1.Rawvideo.Inthiscasetheprocessinginminimal:picturesarecapturedandsent Wealsoinvestigatethreetypesofvideostreamsinordertobetterunderstandtheactual 3.Constantbitrate(CBR)MPEG.Thevideoencoderincludesabuerandratecontroller networkinterface.inthiscasethevideoencoderincludessmallornobuer. directlytothenetworkinterface. boththevideostreamandnetworktypes.theentriesoftable1showthestructureof thisworkbyidentifyingthesectioninwhichthecorrespondingcongurationisstudied.in Section6wesummarizetheobtainedresults. ThevariouscongurationsareshowninTable1.Eachcongurationischaracterizedby inordertoensurethatthebitrateprovidedtothenetworkinterfaceisconstant.inthis caseanencodedpicturemaybedelayedforseveralpictureperiodsbeforeitisforwarded tothenetworkinterface. 2TheModel Inthissectionweintroducethekeyelementsofthesystemdealtwithinthiswork.Wefocus Dealingwiththequalityofvideoandaudioencodingisbeyondthescopeofthiswork.In fact,wemodelthedelayasbeingend-to-end,suchthat,itincludesthejittercompensation areconcernwiththedelayandjitter,thesender-receiversynchronization,andpacketloss. andmediasynchronization.ingeneral,synchronizationisneededforcontinuousplayofaudio onqualityofservice(qos)issuesinadigitalvideoconferencingsystem.inparticularwe andvideostreamsatthereceivingside. 6

RawVideoSection3.1 Dedicated Link Section3.2 Switching Circuit TimeDriven Section3.3 Priority PacketSwitching Asynchronous VBRMPEGSection4.2 CBRMPEGSection5.4 (Section3) (Section4) Section4.5 Section5.5 Section4.3 Section5.6 Section3.4 Section4.4 2.1RequirementsforInteraction (Section5) Table1:CongurationsConsideredinthisWork. Section5.7 Peopleparticipatinginavideoconferenceshouldfeellikebeingface-to-faceinthesamemeeting room.anecessaryrequirementinordertoachievethisgoalistokeeptheend-to-enddelay sinceitissubjecttophysicallimitation,suchas,thespeedoflight... quality,3d,etc..however,theend-to-enddelayis,inourview,themostessentialrequirement forinteractivereal-timeapplications.thedelayrequirementisalsotheleastobvioustosolve, thatthelimitbelowwhichthedelayisnotperceptibleis1ms[13]. (justthinkofhowannoyingarephonecallsroutedthroughsatellitelinks);theassumptionis belowhumanperception.thehumanearissensitivetothedelayresponseinaconversation Requirement1Videoandaudiostreamsaresynchronized. Ofcourse,thedelayrequirementisnotsucient,thereareotherrequirementslikepicture constraintsappliestovideoaswell.thissynchronizationisessentialsincepeopleareusedto Requirement2Theend-to-enddelayoftheaudioandvideostreamsshouldbekeptbelow seeevenbeforehearing(consider,forexample,aspectatorwhoisattendingaconcertbutis farfromthestage).wefocusontheend-to-enddelayintransmissionofvideobecauseitis thehardertokeepbelowthe1mstarget. Sincethevideostreamshouldbesynchronizedwiththeaudiothesameend-to-enddelay 1ms. 7

theaimofreducingthemandkeeptheend-to-enddelaybelowthe1msbound: 2.Network:timetakentomovedatacontainingtheaudioandvideoinformationfrom 1.Processing:itisneededtotransformtheaudioandvideosignalsinaformatsuitable Theend-to-enddelayhasthreemaincomponentsthatwewanttostudyseparatelywith 3.Resynchronization:digitalizationandplayingofbothaudioandvideorequirestricttime fortransmissiononadigitalnetwork.processingmayincludealsothecompressionand decompressionoftheaudioandvideosignals. thesourcetotheotherparticipant(s).italsoencompassesprotocolprocessinginboth senderandreceiver(s). Digitaltransmissionofvideorequiresavideocameracapturepicturesataxedpace(video 2.2RequirementsforVisualQuality addingfurtherdelay;thisisdescribedmoreindetailinthenextsection. andnetworkdelaysvariable,theyintroduceasynchronythathastobecompensatedby synchronizationbetweencaptureofthesignalandreplayingit.beingtheprocessing framerate)andaframegrabberboarddigitalizethem;thetimebetweentwosubsequent capturesiscalledvideoframeperiod.theproducedbitsaresenttothereceiver(s)where theyareconvertedbacktopicturesanddisplayedonamonitoratthesameregularpaceat whichtheyhavebeencaptured.ifthereceiverdoesnotdisplaypicturesatthissamevideo framerate,thereconstructedvideosceneresultscanbeannoying.thisisasortofquality degradationwhichisnotacceptableinthiswork,i.e., Requirement3Thereceivercontinuouslydisplaypicturesattheaverageratetheyhavebeen captured. delayofeachpicturemustbeconstant,i.e.,senderandreceivermustbesynchronized. variability(a.k.a.networkdelayjitter)byintroducinganetworkresynchronizationdelay. correspondingpicture.asshowninfigure1,thebuerusedtocompensatethenetwork Thisisachievedbypossiblybueringthereceivedbitsanddelayingthepresentationofthe jitterisusuallycalledreplaybuer.theoveralleectofthereplaybuer,isthateachpicture Inordertohavecontinuousplayingofthevideoframesonthereceiverside,theend-to-end takesthesametimetotransitfromthesender(pointainfigure1)totheoutputofthe buer(pointbinfigure1);theshorterthenetworkdelayexperienced,thelongerthetime Sincethenetworkdelaycomponentisvariable,thereceiverhastocompensateforthis inmovingfromthepointatothepointbinfigure1isxed.ifapictureexperiencesanetwork thepicturespendsinthereplaybuer(i.e.,thenetworkresynchronizationdelay). Afterthesystem(inparticularthereplaybuer)isdimensioned,thetimeapicturespends 8

End to end Delay becausetheresultingimagequalitydegradationispropagatedtothefollowingvideoframes. atoolargedelay,itcannotbeusedwhendisplayingapicture.thisisnotacceptable,especiallywhenthevideoiscompressedbyeliminatingthetemporalredundancybetweenpictures delaylargerthanthistime,requirement3isnotsatised.ifonlypartofapictureexperiences E A B D andthusitisnotacceptable.hence, Figure1:SynchronizationinVideoTransmission. Network Replay Buffer capture display Ifthisrequirementissatised,thereplaybuercanbedimensionedlargeenoughtocompensateforthenetworkjitterandguaranteesynchronizationmissionveryexpensive,compressionisperformedbeforetransmission.Thistakesavariable amountoftimeaswellastheirdecompression;theencoder/decodersystemintroducesapro- timeisclearlylargerthanthemaximumprocessingdelayexperiencedbyapicture. ofthetimeeachpicturespendsinblockeandinblockddepictedinfigure1isconstant;this cessingresynchronizationdelaythatcompensatesforthisvariability.inotherwords,thesum Sincetheamountofbitsobtainedbydigitalizingapictureisverylargeandtheirtrans- Requirement4Thenetworkdelayisbounded. time time se.infact,awiseresynchronizationmakestheend-to-enddelaybethesumofthemaximum belowaperceptiblebound,inthefollowingwedonotconsiderresynchronizationdelaysper processingdelayandthemaximumnetworkdelayexperiencedbyanypacket.thus,wefocus onthesetwomaximumvaluesinvariousdierentscenarios. tioncomponentsofthesystem.sincetheaimofthisworkiskeepingtheend-to-enddelay Knowledgeoftheresynchronizationdelayisthebasisfordimensioningtheresynchroniza- canbemergedwiththedecoderbuer.intheformercase,thecompensationcanbemore ecient(i.e.,thenetworkresynchronizationdelaybeminimum)becauseitcanexploittiming becompensatedeitherinthenetworkinterfaceorinthedecoderitself,i.e.,thereplaybuer 2.2.1BueringIssues Figure1depictsthereplaybuerasaseparatefunction.Actually,networkdelayvariationcan 9

informationaboutthenetwork.instead,mergingthetwobuersprovidemorerobustness End to end Delay cansometimescompensateeachother. againstmisdimensioningofthebuersbecausethevariationsofnetworkandprocessingdelay E D display,asshowninfigure2.buerscanbepresentinthenetworkinterface(e.g.,ifapacket Figure2:SynchronizationJustBeforeDisplaying. Network switchednetworkisusedincomingbitsmustbeassembledinpacketsbeforetheheaderis Inprinciple,synchronizationcanbeperformedbyasinglebuerinsertedjustbeforethe Buffer capture intheprocessingandnetworkdelays. processed)andinthedecodertostorearrivingdatauntilthereisenoughdatatobeprocessed andwhilethedecoderisbusy.anywaythisbuersarenotusedforcompensatingvariations time fordisplaying,andtaggedwiththedisplayingtime;whenthistimehascome,thepicture beforedecodingthepictures. 2.3TheSystem isdisplayed.thesynchronizationbuerislargerthatthebuersusedforsynchronization Encodedpicturesaredecodedassoonastheyarereceived,bueredintheformatsuitable Figure3.Notallofthemarepresentinanycongurationtakenintoconsiderationinthis Inourview,avideoconferencingsystemiscomposedofthefunctionalblocksdepictedin thevideocamera;thistimecanbeneglectedwithrespecttotheobjective1msend-to-end itintroduces.inthissectiononlycapturecardanddisplayarediscussed;inthefollowing Sectionstheotherblocksaredescribedastheyarerstintroduced. decoderfunctionalitiesarenotincludedinthesystem. work.forexample,whentransmissionofrawvideoisconsidered,boththeencoderand delaybound. Whenapictureisavailableonthereceiversideinaformatsuitablefordisplaying,itis Acapturecard,orframegrabber,takesaround3mstodigitalizeeachpicturecapturedby Inthisworkeachofthesefunctionalblocksisconsideredfromtheviewpointofthedelay insertedintothevideoframebuer.thevideoadaptorperiodicallyscansthevideoframe 1

Encoder Buffer Shaping & Frame Encoder Grabber Packetization buerandtracestheimageonthescreenaccordingtotheinformationstoredforeachpixel. Figure3:HighLevelModelofaVideoconferencingSystem. Network Network Decoder Adaptation Adaptor Theboundonthepresentationdelayis17msforRf=6Hz.Pd=isobtainedby Thedelayintroducedbythedisplaydependsonthevideorefreshingfrequency.Current synchronizingthevideoadaptorwithothercomponentsofthereceiver:thecomponentto monitorsworkatrefreshingfrequenciesrfbetween6hzand1hz,thusintroducinga Pd2";1Rf# Decoder Buffer besynchronizedwillbeidentiedforeachofthesystemcongurationspresentedthroughout Inordertobetteranalyzeeachcomponentoftheend-to-enddelayandshowinwhichcomponentofthesystemitisintroduced,anincrementalapproachisused.Westartfromthe delayisaected. 3TransmissionofRawVideo videoframebuerandwhenthebuerisscannedforrefreshingthedisplay. accordingtotheinstantaneoustimerelationshipbetweenwhenapictureisinsertedinthe thiswork.whennosynchronizationisprovided,thevalueofpdvariesinthegiveninterval Sections4and5weaddfunctionalblockstothecongurationandshowhowtheend-to-end congurationofthevideoconferencingsystemwhichrequiresthesmallestnumberoffunctionalblocksandhasthesmallestnumberofcomponentsintheend-to-enddelay.then,in 11

networkcongurations.althoughvideoisnottransmittedinarawformatduetoitshigh Inthisrstapproachweconsiderthetransmissionofrawvideooveranumberofdierent Figure4:RawVideooveraDedicatedLink Frame Network Network Video aredirectlyconnectedbyalinkhavingcapacityc.thefrbitsofapicturearemadeavailable Figure4showstheblockdiagramofavideoconferencingsysteminwhichsenderandreceiver bandwidthrequirements,itisinstructivetostudythiscase.sinceinthiscasewecanstudy 3.1DedicatedLinkBetweenSenderandReceiver inisolationthenetworkdelayandthepresentationdelay1. Grabber Interface Interface Adaptor dimensionthenumberofbitsencodingapicture. tothenetworkinterface,fromthecapturecard,inafewmilliseconds.thepictureisthen sentwithoutanyadditionalprocessingatthelinkspeed,asshowninfigure5.wecallpicture gatherthebitswhiletheyarebeingreceivedfromthenetwork;assoonasthelastbitofa eachbitexperiencesinthesenderandreceiverbuersanoveralldelaywhichisequaltothe theframegrabber,tothelinkspeedc.abuerisalsoneededonthereceiversideto transmissiondelayoffrbits(dimensionofarawpicture)overalinkofcapacityc,i.e.,fr=c: pictureisreceived,thepicturecanbemovedintothevideoframebuerfordisplay.thus, therstbitspendsthetimeonlyinthereceiverbuer,whilethelastoneonlyinthesender Abuerisneededonthesendersidetomatchtherateatwhichbitsareproducedby buer;intermediatebitsspendtimepartlyinthesenderbuerandpartlyinthereceiverone. wherefristhedimensionofarawpicture,pisthepropagationdelay,andpdisthepresentationdelaywhichdependsontherefreshingfrequencyofthemonitorandthesynchronization relationshipbetweencapturecardandvideoadaptor.pdisnullifthevideoadaptoronthe Thepropagationdelayexistsinallthecongurationspresentedinthiswork,eventhough Ded Raw=Fr C+P+Pd; Ifthetimeneededfordigitizingpicturesisneglected,theend-to-enddelayisgivenby receiverissynchronizedwiththecapturecardofthesender. insomeofthemitisconcealedinsidesomeothercomponentoftheend-to-enddelay.the propagationdelayisaphysicallowerboundontheend-to-enddelayofthesystem;ifsender andreceiver(s)aresofarawaysuchthatp>1ms2,theboundontheend-to-enddelay 1Thetimerequiredfordigitalizing(capture)apictureisconsideredrelativelysmall. 2Propagationdelayof1msinberisabout2,kilometersorhalfoftheEarthcircumference. 12

Bits T 2T 3T Time Rate cannotbesatised(i.e.,face-to-face-likeinteractionisnotpossible),independentlyofthe C congurationusedforthevideoconferencingsystem. thenumberofbitsusedtoencodeeachpixel.forexample,aqcif(quartercommonimage Format)picturehasadenitionof176x144pixels;assumingthateachpixelisencodedon Figure5:GenerationandTransmissionofRawPicturesoveraDedicatedLink. T 2T 3T Time pictureona1mb/slinkis1.98ms:evenconsideringtheworstcaseforthepresentation Frdependsontheresolutionofthepictures(i.e.,thenumberofpixelperpicture)and Production transmission delay,8mscanbespentinpropagationdelayoverthelink,whilekeepingtheend-to-end 8bits,Fr=198kb.InsteadaHighDenitionTeleVision(HDTV)videoframeisdigitalized using192x18pixels,i.e.,162kb.thus,forexample,thetransmissiondelayofaqcif capacity delayunderthe1msbound.instead,anhdtvpicturetakes162mstobetransmitted overa1mb/slink;thisisabovethe1msboundrequiredforinteractionandmoreover itdoesnotallowforreal-timevideo,asshowninthefollowing. Requirement5EachpictureistransmittedwithinthevideoframeperiodT, andthevideoframerate,thecapacityofthelinkmustbelargeenoughtosatisfy Real-timevideotransmissionrequiresthat i.e.,thetransmissiondelaymustbesmallerthant;thus,giventheresolutionofpictures CFr 13T: (1)

Bits T 2T 3T Time Forexample,theminimumlinkcapacityneededtotransmitvideosequencesatHDTV Figure6:GenerationandTransmissionofRawPicturesoveraSlowDedicatedLink. Rate C T 2T 3T Time propagationdelayislarge,thelowerboundonlinkspeedisnotgivenby(1),butbythebound capacitylinkisexploited,thegenerationandtransmissionofpicturesshowsthebehavior depictedinfigure6. mainparameterindeterminingtheend-to-enddelayisthelinkspeed.thehigherthepicture denitionandthevideoframerate,thefasterthelinks.ifthevideoframerateisloworthe denitionis486mb/ssincethehdtvframerateis3picturespersecond.whenaminimum Thus,inavideoconferencingsystemthattransmitsrawvideooveradedicatedlink,the Production ontheend-to-enddelay.thisleadstohighcostforthevideoconferencingsystem,especially videoconferenceisusuallyneeded). whenhighdenitionisrequiredandsenderandreceiverarefar(thatisthesituationinwhich oftrac;inthefollowingotherschemesarestudiedwithrespecttotheresultsshowninthis video.accordingtoothertransmissionschemes,linksaresharedamongvariouscallsortypes particularlyoverlongdistances;nevertheless,giventhespeedoflinks,thecongurationdescribedinthissectionprovidesalowerboundintheend-to-enddelayfortransmissionofraw Thus,dedicatingthewholecapacityofalinktoavideoconferencecallisnotcosteective, Highspeedlinksarenowavailablealsoonlongdistances,buttheyareveryexpensive. Section. 14

isassignedtoeachconnectionusingtimedivisionmultiplexing.thearchitectureofthe Incircuitswitchinglinkcapacityissharedamongdierentcalls.Axedamountofcapacity 3.2CircuitSwitching Figure7:ModelofaVideoconferencingSystemExploitingaCircuitSwitchedNetwork. wereconnectedbyadedicatedlink,eventhoughitsbandwidthisusuallysmallerthanthe videoconferencingsystemisdepictedinfigure7;senderandreceivercommunicateasthey physicalspeedofthelinksonwhichdataactuallyow. Frame Interface spondingfractionofthecapacityofeachlinkonthepathbetweensenderandreceiver.the AvideoconferencecallisallocatedacircuitwithbandwidthBbyreservingthecorre- Grabber Interface Adaptor bitsencodingeachpicturearesentthroughthecircuitatratebasshowninfigure8.this introducesanetworkshapingdelaygivenby: wherepisthepropagationdelayandswistheswitchingdelay.thisend-to-enddelayis (almost)constantbecauseb,p,andswarealmostconstant;thus,networkresynchronization Theend-to-enddelayisgivenby CS Raw=Sn+P+Sw+Pd; Sn=Fr isnotneededinthereceiver. AccordingtoRequirement5forvideoconferencingapplicationSnT,i.e., BFr (3) B,thelargertheamountofbandwidthiswastedbecausethecircuitisbusyonlyforatime SnineachvideoframeperiodT.DuringtheremainingtimeT?Sn,nootherconnectionor Ȯnonehand,thelargerB,thesmallertheendtoenddelay;ontheotherhand,thelarger classoftraccanexploitthisunusedfractionoflinkscapacity. Ifminimumbandwidthisallocatedtothevideoconferencecall,theend-to-enddelayis T largerthanthevideoframeperiod;thelowerthevideoframerate,thelargertheend-to-end 3Mb/s,buttheresultingshapingdelayis67ms.Nevertheless,ifmorebandwidthisallocated delay.forexample,theminimumbandwidthrequiredtosendqcifpicturesat15fps,is B (2)

Bits T 2T 3T Time switchingistosharelinkcapaityinaexibleandecientmanner.however,somepacket becausethecircuitisidleforhalfofthevideoframeperiod.onofthemotivationsforpacket SwitchedConnection. todecreasethenetworkshapingdelayto3ms,5%oftheallocatedbandwidthiswasted Figure8:AmountofBitsandRateinRawVideoEncodingandTransmissionoveraCircuit switchingschemesmayrequiretomaintainlowbandwidthutilizationwhentryingtosatisfy real-timetracrequirements. B T 2T 3T Time 3.3PacketSwitchingwithTimeDrivenPriority Production time:thehigherthelinkspeed,thelongertheidletime.inthissectionweshowhowthelink Figure5showsthatthetransmissionofavideoframecanrequireuseofthelinkforashort canbeusedfortransmissionofotherclassesoftrac(eitherrequiringguaranteesonservice qualityor"besteort")duringitsidleperiods. 3.3.1BasicPrinciplesofTimeDrivenPriority Timedrivenpriority[15]isamultiplexingschemeaimedatsharinglinkcapacitywhileguaranteeingsourcesfromuncontrolleddelays(orevenlosses)duetocontentioninaccessinglinks. andapacketistransmittedduringasingletf. IneachTFaxedamountofbitsTfCcanbesentonalink:dataaregroupedinpackets Thetimeisdividedintimeframes(TFs)ofxeddurationTf(atypicalchoiceisTf=125s). 16

Rate Figure9:GenerationofRawPicturesandMultiplexingwithTimeDrivenPriority. T 2T 3T 4T 5T 6T Rate C sentinasingletf.forthesakeofsimplicity,inthisworkwedonotconsidertheprocessing Figure1:ModelofaVideoconferencingSystemExploitingaPacketSwitchedNetwork. Apictureisdividedintooneormorepartswhichareinsertedintopackets.Eachpacketis T f Frame Packetization Packetization connectedtointermediatesystems,whichinturnareinterconnectedinageneraltopology, introducedbythepacketheaders.inthetfsbetweenthetransmissionoftwosubsequent requiredforpacketizationineitherthesenderorthereceiverandthetransmissionoverhead transmitted(seefigure9),i.e.,tracmultiplexingisperformedonthelinks. picturesofthesamesession(andintheremainingtimeinthesametf),otherdatacanbe Senderandreceiverarenotdirectlyconnectedthroughadedicatedlink;instead,theyare Grabber Adaptor directlyconnected;thepacketheaderidentiesthereceiversothattheintermediatenodes asshowninfigure1.thesendersendsitspacketstotheintermediatenodetowhichitis canrouteittothedestination. mediatelyfollowingtheoneinwhichitwasreceived3.toguaranteethatitisalwayspossible, 3AnodecanhappentorequireanumberofTFstoprocessthepacket;thebestTFfortheforwardingis ThenetworkshowsthebestperformancewhenapacketisforwardedduringtheTFim- 17

BufferFigure11:RISC-likePacketForwarding. C 3 B Output 4 callisnottobeforwarded,theallocatedbitscanbeusedanywaytosendotherdata. othertfs,andeventhenon-reservedbitsinthereservedtfs,canbeusedtosenddatanot TFsarereservedonthelinksonthepathbetweensourceanddestinationandthesource transmitsdataduringthetfsreservedonitsoutgoinglink.atfisreservedforavideoconferencecallwhenafractionofthectfbitswhichcanbeusedduringitiskeptavailablefor forwardingapacket(belongingtothevideoconferencecall)receivedintheprevioustf.the belongingtothevideoconferencecall.moreover,ifapacketbelongingtothevideoconference AreservationofTFsallowsnodestoperformRISClikeforwarding[14]:packetsare A bueredinintermediatenodesforaxedtimeandeachpackettakesaxednumberoftfs 1 2 3 4 5 6 7 8 9 Time tomovefromtheoutputbuerofanintermediatenodetotheoutputbuerofthefollowing tomoveapacketfromsourcetodestinationdependsonlyonthephysicaltopologyofthe oneonthepathtothedestination,asshowninfigure11.asaconsequencethetimeneeded pathandhasadelayvariationorjitterof2tf[15].thisnetworkjitterisxedanddoes notdependontracloadorthepathlengthfromsourcetodestination.asaconsequence, apacketswitchednetworkwithtimedrivenpriorityandresourcereservationcompileswith Requirement4becausenetworkdelayhasapredenedbound.Thebueringtimeineach destination:theend-to-enddelayinthevideoconferencingsystemis 3.3.2TransmissionofVideoandScheduling networkisnotsubjecttocongestionandtherelatedloss. Giventhepaththroughthenetwork,apackettakesatimeLTftotravelfromsourceto nodeisxedatreservationtime:keepingitsmallenoughtoavoidoverowingofbuers,the therstafterthisprocessingtime. 18

canbeeliminatedbysynchronizingthevideoadaptorandthereceivernetworkinterfaceso wherepd2[;1=rf]isthepresentationdelayandrfistherefreshingrateofthedisplay.pd TDP chronizationdelay,i.e.,noreplaybuerisneededonthereceiverside.ldependsonthe continuousplayingrequirement3.hence,thereisnoneedforintroducinganetworkresyn- thatthedisplayisrefreshedjustafterthetfinwhichthepictureisreceived. Thevariationofthenetworkdelayissmallenoughtobetolerated,notwithstandingthe Raw=LTf+Pd ordertohaveintermediatenodesperformingrisc-likeforwarding,thetfsonalinkmustbe numberoftfs:allthenodessharethesameknowledgeoftheordinalpositionofthecurrent TFinsidethetimecycle.Bandwidthisallocatedtoasender/receiverpair,byreservinga numberofhopsonthepath(itisatleastequaltothenumberofhops),theprocessingdelay propernumberoftfspertimecycleoneachlinkonthepathbetweensenderandreceiver.in insideeachnode,andthepropagationdelaybetweeneachpairofintermediatesystems. totransitfromtheoutputbueroftheupstreamnodetotheoutputbueroftheconsidered chosenaccordingtothetfsreservedontheupstreamlinkandthetimeneededforapacket Resourcereservationisbasedonthedenitionofatimecyclewhichencompassesaxed node. thetimecyclebeingequaltothevideoframeperiod;theobtainedresultscanbeextendedto thegeneralcasebytakingintotheconsiderationsmadeinsection3.3.3.atfinthetime cycleisreservedtothevideoconferencecall,asshowninthelowerpartoffigure12.the choiceofthetftobereservedwithinthetimecycleoneachinvolvedlinkiscalledscheduling mustbeanintegermultipleofthevideoframeperiod.forthesakeofsimplicityweconsider anditmustimpactsthemaximumnumberofreal-timeconnectionsconcurrentlysupported Sincerawvideohasanaturalpacingdrivenbythepicturerate,thetimecycleduration bythenetwork[14].inthefollowingtheimpactofschedulingontheend-to-enddelayofthe videoconferencecallispresented;anyway,asschedulingisoutofthescopeofthisworkand itwillnotbeaddressedindetailsinthiswork. thenetworkandtheupperpartshowsthetimingoftheframegrabberdigitalizingpictures.if theframegrabberisnotsynchronizedtothenetworkinterface,thebitsproducedbytheformer arebueredforatimesas ThelowerpartofFigure12depictstheTFsallocatedonthelinkconnectingthesenderto two,sas synchronizationcomponentofthenetworkshapingdelaysinceitisusedtoadaptthetiming delineatetfs,sas atwhichpicturesarecapturedissynchronizedtotheoneusedbythenetworkinterfaceto characteristicsoftheapplicationtotheonesofthenetwork.iftheclockthatdrivesthepace nslowlyincreasesordecreases;whensas nisconstantduringavideoconferencecall.ifthereisadriftbetweenthe n2[;t]whilewaitingtobetransmitted;thisistheapplication overwrittenbyanewone(whensas andwrapsaround,atfcanbeleftunused(whensas nchangesfromtto),andhenceitisnottransmitted. 19 nreachestheboundaryofitsvariationinterval nchangesfromtot)orapicture

Rate Theendtoenddelayisgivenby: Figure12:TransmissionwithTimeDrivenPriority. T 2T 3T 4T 5T 6T Time Rate C S t Time T f TDP Raw=SAS n+ltf+pd (4) EvenwithQCIFformatpictures,capacityoflinksmustbelargerthan1.5Gb/sinorderto CTf.Thiscanbesatisedwhenthelinksarefastenoughsothat Thegivenexpressionholdswhenapicturecanbeincludedinasinglepacket,i.e.,Fr CFr Time Cycle rawvideoframesusingpacketswitchingwithtimedrivenpriorityisgivenby: (e.g.,infigure13ns=2).thus,theend-to-enddelayinavideoconferencingsystemsending sendawholepictureduringasingletf(hdtvdenitionrequireslinkspeedlargerthan 13Gb/s.Wheresuchlinksarenotavailable,apicturecanbesentduringNsTFs,where TDP Raw=SAS n+(nr?1)tf+ltf+pd Ns=&Fr TfC' wherenrnsisthenumberoftfsbetweentherstandthelasttfreservedforthe transmissionofapicture.infact,itcanhappenthatnsconsecutiveavailabletfscannotbe 2

CFigure13:TransmissionwithTimeDrivenPriority. Rate T 2T 3T 4T 5T 6T Time reservedandtheend-to-enddelayisincreased. Rate ThecomponentLTfdependsonthetopologicalandphysicalcharacteristicsofthesystem. foundoneachlinkintheproperpositioninsidethetimecycle;thus,nonconsecutivetfsare WecallnetworkshapingdelaySn=SAS Theend-to-enddelaycanthenbewrittenas4 Time T f TDP Raw=Sn+LTf+Pd (5) Time Cycle generated,tonetworkandtransmissionschemeexploitedtotransmitthem.sndependson thedelayintroducedinordertoadapttherateatwhichthebitsencodingeachpictureare thelinkspeed,thedenitionofpictures,andthescheduling. (whencapturecardandnetworkinterfacearesynchronizedandanoptimalscheduleisfound) UsingQCIFformatat15picturespersecond,Ns=16andSncanvaryfrom1.8ms n+(nr?1)tf problemspresentedinsection3.3.3arise. to68.8ms.consideringascenarioinwhichthepathfromsendertoreceiverincludes3 hops(withnegligiblepropagationandprocessingdelay,i.e.,l=3),sas (C=1Mb/sandPisnegligible)theendtoenddelayis1.98ms,aswasshownin theend-to-enddelayis2.175ms.inthecongurationwithdirectlinkinsimilarconditions 4Ifthetimecycledoesnotcoincidewiththevideoframeperiodandanoptimalscheduleisnotfound,the 21 n=andpd=,

dedicatedlinkcapacityisleftunused. Section3.1.Thus,theend-to-enddelayiscomparableinthetwocases,but97%ofthe turecardandnetworkinterfacewillbemorelikelyandthesas Asequipmentforvideoconferencingapplicationswillevolve,synchronizationbetweencap- sub-optimalschedulecanbeacceptedprovidedthatitcompileswiththe1msend-to-end schedule.thus,iftheroutefromsendertoreceiverischaracterizedbyasmallvalueofl,a apictureissentimmediatelyafterithasbeendigitalized(providedthatanoptimalschedule theloweristhevideoframerate.eventhoughcapturecardandnetworkinterfacearesynchronized,alargenetworkshapingdelaycanbeduetotheimpossibilityofndinganoptimal doesexist).thelackofthissynchronizationcanhaveaworseimpactontheend-to-enddelay ncomponentwillbenullbecause InSection3.3.2isshownhowbandwidthisallocatedtoavideoconferencecallbyreserving ontfinatimecyclewhosedurationequalsthevideoframeperiodt.ifmorethanone 3.3.3MultipleVideoconferenceCallsontheSameNetwork callissubmitted;thisincreasestheprobabilityforthecalltobeblocked. delaybound.iflislarge,anoptimalscheduleisrequiredtothenetworkwhentheconference videoconferencingsystemisexploitingthesamenetworkandtheydonotusethesamevideo framerate,thetimecycleischosenastheminimumcommonmultiplierofallthevideoframe multiplemofitsownvideoframeperiodt.assumingthateachpictureissentwithina periodsinuse. singletf,bandwidthisallocatedbyreservingmtfswithinthetimecycle.ifthetfsare ist,everythinghasbeenstatedaboutthecasem=1,stillholds. evenlydistributedwithinthetimecycle,i.e.,thetimedistancebetweentwosubsequenttf Fromthepointofviewofeachofthevideoconferencecalls,thetimecycleisaninteger ofthepictureinterarrivaltime.ifthisvariationistoolargeaccordingtothecontinuous ofthenetworkdelay;thelessevenlydistributedarethetfs,thelargertheresultingvariation playingrequirement3,thereceivercompensatesitthroughareplaybuerandtheend-toenddelayisaccordinglyincreasedwithrespecttotheonegivenin(4).ifmorethanonetfis usedtosendeachpicture,thechoiceofthetfstobereservedimpactstheend-to-enddelay; IfthereservedTFsarenotevenlydistributed,theirrelativetimedistanceisnotTanymore andthereceiverdoesnotreceivepicturesataconstantpace.thisisequivalenttoavariation videoconferencingapplications.thetwosometimesimposeconictingoptimalitycriteriafor requiringabandwidthreservationtothenetwork,canaskforaschedulethatminimizesthe end-to-enddelayanddoesnotrequirearesynchronizationcapablereceiver,attheriskof schedulingandthusatrade-omustbefoundamongthem[1].avideoconferenceapplication gettingtherequestrefused.alternatively,ifthereceiverisequippedwithareplaybuer adetailedanalysisoftheyieldedend-to-enddelayisoutsidethescopeofthiswork. tocompensatethevariationofthenetworkdelayintroducedbyscheduling,itcanaccepta Ingeneral,schedulingisimportanttobothnetworkperformanceandend-to-enddelayof 22

schedulethatincreasestheend-to-enddelay. multiplesoneoftheother.thelargestvideoframeperiodcanbechosenastimecycleinthe keepmsmall,videoconferencingapplicationshouldchoosevideoframeratethatareinteger network. 3.4AsynchronousPacketSwitching ThelargerM,theharderisfortheschedulingtosatisfytheoptimalitycriteria.Inorderto Thevideoconferencingsystemcanbebuiltonatraditionalasynchronouspacketswitching networkwhichstatisticallymultiplexdataonthelinks.thefrbitsencodingapictureare Assumption1AboundQMonthequeueingdelaydoesexist. insertedinpacketsof(maximum)dimensionpswhicharesentintothenetworkatthefull linkspeed.inthenetworkpacketsexperiencexeddelayduetotransmission(fr=c)and Packetsarequeuedinnetworknodeswhiletheyarecontendingforabusylink;thus,the propagation(p),andvariabledelayduetoqueueing. describedinsection2.2.accordingtorequirement4,thenetworkdelaymustbebound,i.e., queueingdelaydependsontheloadinthenetworkanditdoesnothaveadeterministic Queueingintroducesanetworkjitterthatmustbecompensatedbyareplaybueras 3.4.1NetworkResynchronizationDelay queuedimension;forthispurposetracisshapedattheboundariesofthenetwork. bound.aboundcanbeguessedbuttohavealowprobabilityofbreakingit,itmustbelarge. ontheend-to-enddelay. Moreover,thequeueingdelaycanbekeptsmallerbyavoidingburstsofpacketsthatincrease GiventheboundQMonthequeueingdelay,eachpacketthathasexperiencedashorter queueingdelayisdelayedinthereplaybuer.thisisshowninfigure14:theupperdiagram InthisSectionweanalyzethecontributionofnetworkresynchronizationandtracshaping todeterminethequeueingdelayexperiencedbyapacket.thus,therstpacketreceivedfor showsthearrivaltimeofpicturestothereplaybuer,whilethelowershowstheexittime. Theresultingend-to-enddelayofpicturesis foratime thevideoconferencecallisbueredforatimethatallowsresynchronizationintheworstcase: itisassumedthatthepackethasexperiencedtheminimumqueueingdelayqmandisbuered Ifthenetworkinterfacesofsenderandreceiverarenotsynchronized,thelatterisnotable Async Raw=Fr C+P+QM+Pd: (6) 23

F F r Q r + P + Q + P + Q C M C M+ T Figure14:NetworkResynchronizationDelaytoCompensateQueueingDelayVariation. F Time F F r r + P + Q C m C + P + Q m + T F r r + P + Q C M+ T C + P + Q m + 2T Q Q M m F delay F r r + P + Q + P + Q C M C M+ T followingpacketsduetoqueueingandresynchronizationisbetweenqmandqm+q.ifthe packet,delaysitbyqaccordingto(7).asaconsequence,theoveralldelayexperiencedby Thefollowingpacketsareresynchronizedaccordingly,asshowninFigure14,becausetheyare exitingincasethereceiverdoesnotknowtheactualqueueingdelayexperiencedbytherst experiencesanactualqueueingdelayqm.themiddlediagramshowsthetimingofpackets retrievedfromthereplaybuerattheconstantpaceatwhichpicturesaredisplayed. TheupperdiagramofFigure15showsaresynchronizationexamplewhentherstpacket Time receiverknowstheactualdelayexperiencedbytherstpacket(e.g.,senderandreceiverhave acommontimereferenceandthepacketcontainsatimestampindicatingwhenitwassent), thelowerdiagraminfigure15. doesnotdelayitatallandtheexitingtimefromthereplaybueraretheonesdepictedin callanddependsonthequeueingdelayexperiencedbytherstpacketreceived,er=when whereer2[;q]istheexcessresynchronizationdelay;itisconstantoveravideoconference senderandreceivernetworkinterfacesaresynchronized. Insummary,theend-to-enddelayisgivenby Async Raw=Fr C+P+QM+Er+Pd 24 Q=QM?Qm (7)

Q F r C + P + QM F r C + P + Q M+ T F r C + P + Q m F r C + P + Q m + T F r C F r + P + Q M+ T C + P + Q m + 2T Figure15:NetworkResynchronizationDelaytoCompensateQueueingDelayVariation. Time F r C + P + Q M+ Q Q Q M m Q F r + P + Q C M F r C F r C + P + Q M+ T 25 + P + Q M+ T + Q Resynchronization delay Resynchronization delay Time Time

Figure16:ModelofaVideoconferencingSystemExploitingaPacketSwitchedNetworkwith TracShapingatNetworkBoundaries. 3.4.2TracShapingatNetworkBoundaries ResourcesmustbeallocatedintothenetworkinordertoguaranteetheboundQMonthe Sincetransmissionoflargeamountsofdataatlinkspeed(bursts)causessuddengrowthof queuingdelay(andpossiblylimittheamountofdatalostinthenetworkduetocongestion). queues,largeamountofresourcesisreservedforburstysources:limitsqueuesgrowthby keepinglinkutilizationlow. Frame Packetization Packetization Grabber Adaptor widthbetweensenderandreceiverwhilekeepingtheburstinessbelowapredenedlevel. Assumption2Tracshapingisusedatthenetworkboundarytoprovideanaverageband- Atokenisremovedfromthebucketforeachdataunitsent.Weassumetoexploitabuered asaleakybucket[2],asshowninfigure16.theburstytracgeneratedbyasourceisshaped longasthenumberoftokensinthebucketislargerthanthepacketsdimension(indataunits). byaleakybucketintoaowhavingaverageratebandcontrolledburstiness.abucketis packetsarepresentedtothetracshaper,theyaresentintothenetworkatlinkspeed5as assumedtocontainatmostatokensanditislledattherateofbtokenspersecond.when Thesourceshapesthetracitisgeneratingbyusingatracshapingmechanismssuch leakybucketwhichismodeledasshowninfigure17.whenthebucketisempty,packetsmay delay.accordingtorequirement5,aswearedealingwithreal-timevideotransmission,the andtheresolution,thetraccharacteristicsofarawvideosequenceareknowninadvance, shapedtracattheexpensesofanetworkshapingdelay.since,giventhevideoframerate thebuercanbedimensionedproperlysothatitneveroverows. thebuckettosendthepacket.thisbueradaptsthesourceratetothecharacteristicsofthe bedroppedorbuered(alsoknownasbueredleakybucket)untilenoughtokensarepresentin timetakentorellthetokenbucketmustbesmallerthanthevideoframeperiod,i.e., Thechoiceoftheparametersofthetracshaperdeterminesitseectsontheend-to-end Suchatracshapercanbebuiltbycascadingtwosimpleleakybuckets. asitiscaptured,asshowninfigure18wherea=ps.tracshapingdoesnothaveanyeect 5Thereisavariantcalleddualleakybucketthatsendsburstofdataataratelowerthanthelinkspeed. IfAFrthewholepictureisinsertedinpacketswhicharesentatthelinkspeedassoon BFr 26 T: (8)

andtheaveragerateisfr=t,asdeterminedbythevideoframerate(1=t)anddimension (Fr),andtheend-to-enddelayisgivenby(6). Figure17:ModelofaLeakyBucket. B A frame): Fr?Atokens(since,thanksto(8)thetokenbucketisfullbeforestartingtransmittinga showninfigure19wherea=ps;afterwards,thetimeneededtogeneratepstokens(i.e., Ps=B)iswaited.Thus,thenetworkshapingdelaySnisgivenbythetimeneededtogenerate IfA<Fr,A=PspacketsofdimensionPsaresentassoonasthepictureisdigitalized,as Unshaped Shaped Traffic Sn=Fr?A Traffic siondelayofapacket(ps=c),theend-to-enddelayisgivenby: ThetimeneededtotransmitapicturebeingthenetworkshapingdelaySnplusthetransmis- wherepisthepropagationdelayonthelinksonthepathbetweensenderandreceiver,qm maximum,thenetworkshapingdelayisnotpresent.nevertheless,thelackoftracshaping isthemaximumqueueingdelay,ander2[;q]istheexcessresynchronizationdelay. atnetworkboundariesresultsislowerlinkutilizationorlargermaximumqueueingdelayqm. Comparing(6)and(1)themaindierenceisthatintheformer,wheretheburstinessis Async?Sh Raw=Sn+Ps C+P+QM+Er+Pd (1) queueingpolicyusedbynodes.forexample,ifnetworknodesareperformingweightedfair queueing,theboundisinverselyproportionaltotheallocatedbandwidth[3]: Themaximumqueueingdelaydeliveredbythenetworkdependsonthereservationand QM=o1B 27 B (9)

Bits Figure18:HighBurstinessTransmissionofRawVideooverPacketSwitchedNetworks. T 2T 3T Time Rate C B T 2T 3T Time Production Bits Figure19:LowBurstinessTransmissionofRawVideooverPacketSwitchedNetworks. T 2T 3T Time Rate A 28 A C B C B T 2T 3T Time A A Production B C

ofbitsneededtoencodeeachpicturebyexploitingspatialandpossiblytemporalredundancy Transmissionofrawvideorequireslargeamountofbandwidth,particularlyiftheend-to-end delayhastobekeptunderthe1msbound.thus,compressionisusedtoreducetheamount 4TransmissionofVBRMPEGVideo TheMotionPictureExpertGroup(MPEG)[11,25,8]videoencodingstandardwasdesignedfordigitalstorageofqualityvideoforlaterre-playing.Adigitalizeduncompressed 4.1MPEGOverview turesaresentoverthenetworktothereceiverwheretheyaredecodedanddisplay. presentinsidepicturesandbetweensubsequentpictures,respectively.thecompressedpic- 4.1.1Intra-frameCoding digitalizedpicturesandencodeseachoftheminoneoftwodierentways6. (n=2)x(m=2)pixelsofchrominanceinformation.theencoderreceivesininputasequenceof imageiscomposedofonematrixofnxmpixelsofluminanceinformationandtwomatricesof luminanceandthetwochrominancecomponents)isdividedin8x8pixelblocks.oneach coecientsofthespatialfrequenciespresentintheblock. blockadiscretecosinetransform(dct)isperformed;ityieldsan8x8matrixwiththe Spatialredundancyinsidethepictureisexploitedtogaincompression.Thepicture(the neededtorepresentitthankstothefactthatmanycoecients(particularlythehighfrequency moresensitivetohighspatialfrequencies[26]:highfrequencycoecientsarequantizedmore Actually,adierentquantizationstepsizeisusedforeachcoecientbecausehumaneyeis coarselythanlowfrequencyones,buttheperceivedvisualqualitydoesnotdegrade.thus, eachcoecientismultipliedbythecorrespondingelementofan8x8quantizationmatrix. Eachcoecientisquantizedbyintegerdividingitbyanintegercalledquantizationstepsize. ones)arezero.finally,therun-lengthencodedsymbolsequenceishumanencoded.the 4.1.2PredictiveCoding pictureencodedinthisfashioniscalledi-frame. Theresultofthequantizationisrun-lengthencodedinordertominimizethenumberofbits withrespecttoi-frames.thepictureisdividedintomacroblocks(mbs),eachcomposedof Temporalredundancyiseliminatedtoreducethenumberofbitsneededtoencodethepicture, periodsthatweconsidernotacceptableiftheend-to-enddelaybekeptbelow1ms.thus,thiscompression techniqueisnotconsidered. becoded,areferencesubsequentpicturemustbecapturedandcoded.thisintroducesadelayofsomeframe 6Actually,athirdtypeofencoding,calledbidirectionalpredictivecoding,exists.Inorderforapictureto 29

thetwochrominancecomponents.givenambtoencode,motionestimationisperformed, a16x16pixelmatrixofluminanceinformationandthetwocorresponding8x8pixelblocksof timessmallerthani-frames.thesmallerthedierencebetweenthembbeingcodedand DCT,quantization,run-lengthencoding,andHumanencoding.Ifa\similar"MBisnot found,eachblockofthembisencodedlikeablockinani-frame. dierencebetweentheactualmbandthereferenceoneiscalculatedandcodedbyperforming i.e.,thepreviouspictureissearchedfora\similar"mb.ifthismbisfound,thepixelbypixel thereferenceone,thehigherthecompressiongainedthroughdctandrun-length/human periodtotheother. encoding.themoresimilartwosubsequentpictures,thehigherprobabilityofndingamb \similar"totheonebeingcodedandthusyieldingbettercompression;subsequentpictures aresimilarifthesceneisslowmovingbecauseifdoesnotchangemuchfromavideoframe PicturescodedinthisfashionarecalledP-framesandtheirdimensionisfrom2to4 AvideosequenceiscompressedbyencodingapictureoutofNasI-frame,andtheremaining 4.1.3BitGenerationRateandQuantization N?1asP-frames;thesequencestartingwithoneI-frameandterminatingwiththelast consecutivep-framesiscalledagroupofpictures(gop).thelargern,thesmallerthe overallamountofbitsusedtoencodeascene.nevertheless,ifthenetworkintroducesan toerrorsthestream. isencodedwithinthevideoframeperiod,butap-frameisfrom2to4timessmallerthan picturenotaectedbytheerrorandstopsitspropagation,i.e.,thelargernthemoresensitive errorinanencodedpicture,itpropagatesinthefollowingones;thenexti-frameistherst videosequenceswitharesolutionof72x48pixels;animagefromeachsequenceisshownin ani-frame.moreover,evenvideoframesofthesametypehavedierentdimension:the ofbitsproducedbythesoftwarempegencoderdvdenc[17]duringtheencodingoffour dimensionofani-framedependsonthespatialredundancypresentintheimage,whilethepframeonedependsalsoontheamountoftemporalredundancy.figure2showstheamount Therateofthebitstreamproducedbytheencoderhashighvariability:eachpicture (motion)redundancyineachscene. atallfrequenciesandcoarsequantizationoflowfrequenciesismorenoticeabletohumaneye Figure21.Theoverallnumberofbitsencodingeachsequenceandthewayitisdistributed frameorpredictive)beingperformedonit.intra-framecodedmbstypicallycontainenergy thanhighfrequencies.instead,predictivecodedmbshavehigherenergyathighfrequencies amongthepicturesaredierentduetothedierentamountofspatial(detail)andtemporal zones[26].thus,theactualquantizationstepsizeisobtainedbymultiplyingthequantization andhumaneyeislesssensitivetocoarsequantizationofpredictivecodedmbs. AdierentquantizationmatrixisusedforaMB,dependingonthetypeofcoding(intra- Moreover,humaneyeismoresensitivetoquantizationerrorinatareas,thanverydetailed 3

Cheerleaders Hockey 5 5 Picture dimension (Kb) 4 3 2 1 Picture dimension (Kb) 4 3 2 1 3 6 9 12 15 3 6 9 12 15 Bikes 5 Picture dimension (Kb) 4 3 2 1 Figure2:NaturalDimensionofMPEGEncodedPictures. Table tennis 5 31 3 6 9 12 15 Picture dimension (Kb) 4 3 2 1 3 6 9 12 15

Figure21:ImagesfromtheFourVideoSequencesUsedintheExperiments. 32

matrixbyamblocalactivityfactorthatchangesonambbymbbasiswiththegoalof bitsproduced.thewayinwhichthemblocalactivityfactoriscalculatedisdependenton theimplementationoftheencoderandisbasedontheestimationofthe\activity"levelof themb[9][26]. basis,providesoverallcontrolonpicturedimension.theproductofthemblocalactivity keepingconstantthevisualqualitythroughoutthepicturewhileminimizingthenumberof factorpmbandtheglobaldistortionlevelfactorggivesthequantizationparameterqmbfor MBmb,i.e., Moreover,aglobaldistortionlevelfactorthatpossiblychangesonapicturebypicture wherecmb Insummary,quantizationisperformedaccordingtothefollowingformula: i;jisthe(i;j)dctcoecientofmbmb,^cmb i;j=cmb Qmb=pmbG qfijpgi;j i;jqmb thereisasmalldierencebetweenthedimensionofi-framesandp-frames.infact,ononeside, keptconstantoverthewholevideosequence.thedimensionyieldedforthepicturesisplotted infigure22:thelargerg,thelargerthepictures.forsmallvaluesofg(nerquantization), mb,andqfijpg accordingtothetypeofencodingbeingperformedonthecurrentmb. FoursequenceswereencodedusingfourvaluesfortheglobaldistortionlevelG,whichwas i;jiselement(i;j)ofthequantizationmatrixusedfori-frameorp-framecoding, i;jisitsquantizedvalue,qmbistheformb arenotreducedtozerobythenequantization.ontheothersidethehighfrequencydct P-framesdimensiongrowsbecausethesmallDCTcoecientsofdierentiallyencodedMBs coecientsofintra-framecodedmbsarecoarselyquantizedbythecorrespondingelements ofthequantizationmatrix.thisisconrmedbytheplotsinfigure23whichshowthatfor frames.figure24plotstheaveragequantizationparameterusedoneachpicture;eventhough constant.thevariationislargerforlargervaluesofgbecausegampliesbmbvariations. G=1theSignaltoNoiseRatio(SNR)duetotheencoding7issmallerforI-framesthanforP- itchangesduetothecharacteristicsofthepictures(i.e.,becausebmbchanges),itisalmost qualitybecauseitdoesnottakeintoaccountthecharacteristicsofthehumanvisualsystem.thus,even thoughcoarsequantizationofhighfrequencycomponentsintroducesanerror,itdoesnotnecessarilydegrade theperceivedvisualquality. mathematicalmeasureofthequalityofapicture,butitdoesnotexactlycorrespondtotheperceivedvisual intoaccount. VBRMPEGcompression;theimpactofdierenttransmissionschemesonthedelayistaken 7TheSNRiscalculatedaccordingtotheformulareportedinSectionB.4.TheSNRisusedtoprovidea Inthefollowingwestudytheend-to-enddelayinavideoconferencingsystemthatuses 33

3 Cheerleaders 3 Hockey Picture Dimension (Kb) 25 2 15 1 5 Picture Dimension (Kb) 25 2 15 1 5 Picture Dimension (Kb) 3 25 2 15 1 5 3 6 9 12 15 3 6 9 12 15 G = 1 G = 1 G = 15 Bikes G = 2 Table Tennis 3 Figure22:DimensionofPicturesCodedaccordingtoMPEGStandard. 25 2 15 1 5 34 3 6 9 12 15 3 6 9 number number 12 15 Picture Dimension (Kb)

Cheerleaders Hockey 5 5 SNR (db) 45 4 35 SNR (db) 45 4 35 3 3 3 6 9 12 15 G = 1 G = 1 G = 15 Bikes G = 2 3 6 9 12 15 Table Tennis SNR (db) 5 45 4 35 3 Figure23:SNRofVBRMPEGEncodedVideoSequences. 5 45 4 35 35 3 3 6 9 12 15 3 6 9 12 15 SNR (db)

Average Quantization Parameter 2 18 16 14 12 1 8 6 4 2 Cheerleaders Average Quantization Parameter 3 6 9 12 15 3 6 9 12 15 G = 1 G = 1 G = 15 Bikes G = 2 Table Tennis 2 2 Figure24:AverageQuantizationParameterUsedforVBREncodingaccordingtoMPEG 18 18 Standard. 16 16 14 14 12 12 1 1 8 8 6 6 4 4 2 36 2 3 6 9 12 15 3 6 9 12 15 Average Quantization Parameter Average Quantization Parameter 2 18 16 14 12 1 8 6 4 2 Hockey

Figure25:VideoconferencingSystemCongurationwithDedicatedLinkbetweenSenderand Receiver. Frame Network Network Encoder Decoder Grabber Interface Interface Adaptor 4.2DedicatedLinkBetweenSenderandReceiver Processed Encoder Output link,asshowninfigure25.theencoderprocessespicturesandproducesthebitsencoding Minimumdelaycanbeobtainedifsenderandreceiveraredirectlyconnectedbyadedicated Bits themasi-framesorasp-frameswiththetimingshowninfigure26:thetimetakentoencode Figure26:BitProductionofaNaturalVBRMPEGEncoder. apictureisnotconstantaswellastheamountofbitsproduced. suitableforthevideoadaptor. bitscanbesentoverthededicatedlinkassoonastheyareproduced.afterapropagation delayp,bitsgettothedecoderwhichdecodesthestreamgeneratingpicturesintheformat Inprinciple,ifthelinkspeedishigherthanthenaturalinstantaneousrateoftheencoder8, (I) T 1 (P) 2T 2 (P) 3T 3 (P) 4T 4 (I) 5T Time eventhoughitismorepredictableanditsvariationsaresmaller.thedecoderkeepsconstant oftheencodinganddecodingtimeandintroducestheprocessingresynchronizationdelay. playing)bybueringtheencodedordecodedpictures.thiscompensatesforthevariability thedelaysynchronizationbetweentheinputoftheencoderanditsownoutput(continuous Thetimeneededtoencodeapictureisnotconstant;alsodecodingtimeisnotconstant insteadofd. bitsitisreceiving.forthepurposeofthisworkweconsiderconstant9thedecodingdelayd. availabilityofspecialpurposehardware,andonhowearlythedecoderstartsprocessingthe 8ThisislikelybecausethepeakrateoftheencoderisoftheorderoffewtensofMb/s. 9Ifthedecodingdelayisnotconstantthedecodercompensatesitsvariationsand(11)holdswithmaxD Thedecodingtimestronglydependsonthedecoderimplementation:i.e.,bothonthe 37

wherecmisthemaximumcodingdelay;whenaframetakeslessthancmtobeencoded,it dealingwithreal-timevideo Theend-to-enddelayisgivenby Ded Requirement6Apictureshouldbeencodedwithinthevideoframeperiod, isdelayedbytheremainingtimeinthedecoderbuer. Thefastertheencoderanddecoder,thesmallertheend-to-enddelay;infact,sinceweare VBR=CM+P+D+Pd (11) tothehighestframeratetheencodersupports.thus,ifasceneiscapturedat3frames i.e.,cmt.themaximumencodingdelayisusuallythevideoframeperiodcorresponding persecondandthepropagationdelaybetweensenderandreceiveris2ms(e.g.,theyare Theyieldedcompressionislowerandalargernumberofbitsisneededtoencodeavideo connectedthroughatransoceaniclink),theend-to-enddelayisaround6ms,i.e.,acceptable toguaranteeinteractivitybetweentheparticipantsinavideoconference. consequentlytheend-to-enddelay)canbereducedbyusingonlyintra-encoding(i-frames). sequence. to-enddelayobtainedinavideoconferencingsystemexploitingmpegcompression.never- theless,itisnotecientgiventheexpensivenessofbandwidth:infact,thecapacityofthe ThesystemcongurationdiscussedinthisSectionprovidesalowerboundontheend- Motionestimationisthemosttimeconsumingpartoftheencodingprocess:EM(and howtheend-to-enddelayisaectedbytheusageofmultiplexingschemestomoreeciently exploitbandwidthoflinks. linkconnectingsenderandreceiverisonlypartiallyused.inthefollowingsectionswestudy 4.3PacketSwitchingwithTimeDrivenPriority wherelisthenumberoftfsapackettakestotravelfromsendertoreceiverandpdisthe ingresslink,asshowninfigure28.theend-to-enddelayofthesystemisgivenby producedbytheencoder,theyareinsertedintoapacketsandsentatthefullspeedofthe encodedpictures,asshowninfigure27.assoonasallthebitsencodingapictureare Apacketswitchednetworkwithtimedrivenprioritycanbeexploitedfortransmissionof presentationdelay.forthesakeofsimplicity,inthisworkweconsiderthattheprocessing requiredforpacketizationdoesnotintroducedelay.expression(12)istheactualend-to-end delayonlyifthenodesonthepathfromsendertoreceiverareabletoperformrisc-like forwardingofpackets(seesection3.3.1).38 TDP VBR=CM+LTf+D+Pd

SenderandReceiver. Figure27:VideoconferencingSystemCongurationwithPacketSwitchedNetworkbetween Frame Packetization Packetization Grabber Encoder Decoder Adaptor Bits Rate Figure28:TimeDrivenPrioritywithoutReservation. T 2T 3T 4T 5T 6T T f 39 Time Time

mustbeallocatedinthenetworkandvideoframessentduringreservedtfs.toreserve mensionofthepicturebeingsent.aswasshowninfigure2,thenumberofbitsencodinga resourcesinpacketswitchednetworkswithtimedrivenpriority,theamountofdatatobe capacitycanbereservedduringthepropertfs. sentandtheirtimingmustbeknownatreservationtimesothattheproperfractionoflink ToguaranteethexedboundLTfonthenetworkdelayandlossfreedelivery,resources tioncannotbeaccurate.ifduringatfausersendsmorebitsthantheamountreservedto picturedependsonthetypeofcodingusedforthecorrespondingpicture,andthespatialand possiblytemporalredundancypresentintheimage.sincetheamountofredundancydepends onthescenebeingencoded,picturedimensionisnotknowninadvanceandresourcereserva- it,thenetworkdoesnotprovideanyguaranteeonthedeliveryoftheexcessbits.if,onthe Theamountofbitsreservedshouldbelargerthan(andascloseaspossibleto)thedi- otherhand,thevideoconferencingapplicationusesonlyafractionofthereservedcapacity, theleftoverbandwidthcanbeusedbybesteorttracandisnotwasted.eventhoughthis isacceptablefromthenetworkpointofview,thesolutionisnotoptimalfortheuserthatis forbothtypesofpicturesgiventhebandwidthtobeallocatedtoavideoconferencecall.then, possiblypayingfortheallocatedbandwidthandwouldliketouseitallbyhimself. theend-to-enddelay.eventhoughsomecongurationscandeliverunacceptableend-to-end andnetworkinterface,andthescheduling(i.e.,thechoiceofthetfstobereserved)impact weshowhowtheencodingprocesscanbetunedinordertocontrolpicturedimensionso thatthevideoconferencingsystemneverusesmorebandwidththantheamountallocatedand exploitsasmuchofitaspossible.lastly,weanalyzehowthesynchronizationbetweenencoder InthefollowingofthisSectionweshowhowtodeterminetheamountofbitstobereserved delay,ifthesystemisadequatelyequippedandoperated,itsperformanceisactuallygiven by(12). 4.3.1ChoosingaBoundonPictureDimension andp-frames.theseamountsdeterminethebandwidthreservedtothevideoconferencecall AsexplainedinSection4.1.2,theslowerthemotioninthescenebeingencoded,thelargerthe as slowmovingscenes,weproposetoreservedierentamountsofbitsfortransmissionofi-frames dimensionofi-frameswithrespecttop-frames.sincevideoconferencesareexpectedtobe isthenumberofpicturespergop,andntisthegopduration. anaturalmpegencodedependsontheamountofmotioninthescene.thepictureratio wherefiandfparetheamountofbitsreservedfori-framesandp-frames,respectively,n AsdiscussedinSection4.1.3,therelativedimensionofI-framesandP-framesyieldedby Btdp=FI+(N?1)FP (13) 4

encodedandtransmitted.thisisingeneraladiculttask,butintheparticularcaseof mustthenbechosenwiselydependingtotheamountofmotionexpectedinthescenetobe Figure29:MPEGEncoderforControllingDimensionofFrames. Base Bounded Encoder Picture videoconferencingapplication,scenesarelikelytobeslow. beexpressedasfunctionofthebandwidthtobeallocatedas CombiningEquations(13)and(14),theamountofbitstobereservedtoeachframecan =FI Quantization I frame parameter Rate Bound (FP=BNT P frame Bound 4.3.2ControllingDimensionofEncodedPictures Thereservedbandwidthisusedmoreecientlyiftheencodingprocessiscontrolledandthe numberofbitsencodingeachpictureiskeptbelowtheamountofbitsreservedinthetf duringwhichthepictureissent.thisamountsareupperboundpicturedimensionandcan N+?1 mostsuitedtothispurpose.forexample,figure22showsthatthedimensionofbothi-frames encoderisextendedwitharatecontrolfunction.ittunestheparametersofthebasicmpeg codingprocesssothatthedimensionofeachpictureisbestttedtothetargetassociatedto beusedtodeviseatargetdimensionforeachtypeofvideoframe.asshowninfigure29,the Nevertheless,Gisnotsuitableforourpurposeofcontrollingpicturedimensionbecauseits itstype. andp-framesisstronglydependentonthevalueoftheglobaldistortionlevelg(whichisoneof thecomponentsofthequantizationparameter):thelarger,thesmallertheencodedpictures. valuemustbechosenbeforestartingtheencodingofthepicture.thereexistproposalsto predicttheamountofbitsanencodingprocessisgoingtoproducebasedontherawscene[22] AmongtheparametersoftheMPEGencodingprocessthequantizationparameteristhe orontheportionofstreamalreadyproduced[12].however,togetserviceguaranteesfrom thenetwork,thetargetmustneverbeexceededandanadaptiveapproachbasedonfeedback fromtheoutputoftheencodermustbeexploited. 41 FP (14) FI=FP (15)

produced.theratecontrolalgorithmalsoproposeshowtore-quantizethepictureifthe Thiscanleadtocodingtimesnotacceptableforreal-timeencodingasrequiredinvideoconferencingapplications(Requirement6).[5]proposestoexploitarate-quantizationmodelto yieldeddimensionisnotcompliantwiththetarget. chooseg.themodelistunedaccordingtothecharacteristicsoftheencodedstreamalready Otherauthorsproposeiterativeapproaches[28,16]todetermineasuitablevalueofG. parameterforeachmbas bmbcanbechangedonambbymbbasisinordertoadaptivelytunethequantization Alternatively,theMBlevelratecontrolfactorbmbisusedforcalculatingthequantization quality.manyapproacheshavebeenproposedintheliteratureforuniformlychoosingthe 17].Varyingthegranularityofquantizationthroughoutthepicturedeliversnonuniformvisual accordingtothetargetandthefeedbackobtainedbymonitoringthestreambeingproduced[9, Qmb=Gpmbbmb Thus,wehaveperformedsomeexperimentstoprovethatpicturedimensioncanbecontrolled ratecontrolfunctionwhichimplementsthealgorithmshowninfigure3. visualsystem[26]. astheproposednetworkrequires.thesoftwareencoderdvdenchasbeenexploited;ithasa quantizationparameter[1]possiblytakingintoaccountthecharacteristicsofthehuman Theforegoingapproacheshavebeenproposedandanalyzedinscenariosdierentfromours. (lack)ofbitsandthetoleranceonthetarget.theaverageofthemblevelcontrolfactor andthetotalnumberofmbsmb.theincrement(decrement)isproportionaltotheexcess exceeds(isbelow)theexpectednumberofbits.thisiscalculatedaccordingtothetargetft isincremented(decremented)withrespecttobmb?1iftheamountofbitsfproducedsofar overthewholepictureisusedtocalculatetheglobaldistortionlevelforthenextpictureof Ftandatoleranceonit(intermsofmaximumandminimumacceptabledimensions).bmb ThefunctionwhichcalculatestheMBlevelratecontrolfactorbmbisprovidedwithatarget ofyieldinguniformvisualqualitythroughoutpictures.nevertheless,theobjectivehereisto nottodeviseanoptimalmethodtocontrolthedimensionofframes. thesametype.theexperimentaldatashowninfigures3136areobtainedusingg=15for showthatthedimensionofframescanbekeptbelow(andverycloseto)apredenedbound, boththersti-frameandtherstp-frame.thisalgorithmisnotwisefromthepointofview pictureratios(oneineachrowofthefigures). theshownbounds(fminthealgorithminfigure3):theboundsareneverexceeded.the boundsaredeterminedaccordingtoequation(15)assumingthreebandwidthamountsand twocomplementaryreasons: AsshownbyFigures33and34,theaveragequantizationparameterhashighvariationfor Figures31and32plotthepicturedimensionobtainedencodingfourdierentsceneswith 42

accordingtothetypeofthepicturetobeencoded,define: F=numberofbitsencodingMB1withquantizationparameterQ1=p1GfIjPgb1 formb=2tototalnumberofmbsmb b1=1 choosegfijpgaccordingtothetype(i-frameorp-frame)ofcoding amaximumdimensionfm(reservednumberofbits) atargetdimensionft(e.g.,5%belowreservednumberofbits) aminimumdimensionfm(e.g.,1%belowreservednumberofbits) elseiff<ft iff>ft elsebmb=decr(f;mb;fm;ft;mb) iff?ft bmb=maxb Mbmbthen Mbmb>FM?Ftthen Qmb=pmbGfIjPgbmb elsebmb=bmb?1 elsebmb=incr(f;mb;fm;ft;mb) ifft Mbmb?F>Ft?Fmthen bmb=minb Mbmbthen GfIjPg=GfIjPg F=F+numberofbitsencodingMBmbwithQmb MbPMb Figure3:VideoFrameDimensionControlAlgorithm. mb=1bmbaccordingtothetypeoftheencodedpicture 43

18 Cheerleaders - Picture ratio 2 18 Hockey - Picture ratio 2 Picture dimension (Kb) 15 12 9 6 3 Picture dimension (Kb) 15 12 9 6 3 Picture dimension (Kb) 18 15 12 9 6 3 3 6 9 12 15 4 Mb/s 8 Mb/s Cheerleaders - Picture ratio 3 16 Mb/s 18 Picture dimension (Kb) 15 12 9 6 3 3 6 9 12 15 Hockey - Picture ratio 3 Picture dimension (Kb) 18 15 12 9 6 3 3 6 9 12 15 3 6 9 12 15 I-frame bound Cheerleaders - Picture ratio 4 P-frame bound Hockey - Picture ratio 4 18 Figure31:DimensionofPicturesinControlledMPEGEncoding. 44 15 12 9 6 3 3 6 9 12 15 3 6 9 12 15 number number Picture dimension (Kb)

18 Bikes - Picture ratio 2 18 Table Tennis - Picture ratio 2 Picture dimension (Kb) 15 12 9 6 3 Picture dimension (Kb) 15 12 9 6 3 Picture dimension (Kb) 18 15 12 9 6 3 3 6 9 12 15 4 Mb/s 8 Mb/s Bikes - Picture ratio 3 16 Mb/s 18 Picture dimension (Kb) 15 12 9 6 3 3 6 9 12 15 Table Tennis - Picture ratio 3 Picture dimension (Kb) 18 15 12 9 6 3 3 6 9 12 15 3 6 9 12 15 I-frame bound Bikes - Picture ratio 4 P-frame bound Table Tennis - Picture ratio 4 18 Figure32:DimensionofPicturesinControlledMPEGEncoding. 45 15 12 9 6 3 3 6 9 12 15 3 6 9 12 15 number number Picture dimension (Kb)

1.Thevariationofpicturedimensionislimitedbytheparametersprovidedtotherate controlfunctionintheexperiments:thetarget(ftinthealgorithmshowninfigure3) SNRplottedinFigures35and36.Theratefunctioncontrolisnotdesignedtokeepthe 2.Theratecontrolfunctiondoesnothavetheobjectiveofdeliveringuniformquality Asaresultthequalityofpicturesisnotuniformthroughouteachscene,asshownbythe (Fm)1%smallerthanthetarget. hasbeenset5%belowtheupperbound(fm)andthelowerboundofpicturedimension qualityuniforminsidepicturesaswell.asaconsequence,themblevelratecontrolfactor hashighvariabilitywithineachframe. Thescenesusedintheseexperimentshavepicturesmoredetailedandahigheramount throughoutthescene. functionwoulddelivermoreuniformqualitywhendealingwithwithtypicalvideoconferencing ofmotionthantypicalvideoconferencingheadandshoulderscenes.thesamesimplecontrol choiceofthetfsinwhichthebitsarereservedaectstheend-to-enddelay;theconditions underwhichitcanbereducedtoexpression(12)arealsopresented. scenes.moreover,enhancingtheratecontrolfunctiontoreducethequalityvariationinthis scenarioisnothard. 4.3.3SynchronizationbetweenEncoderandNetwork allocatedbyreservingxedamountofbitsforthetransmissionofpicturesofthesametype. TheamountofbitsiscalculatedusingEquation(15).Inthefollowingweanalyzehowthe Havingtheratecontrolfunctionboundingpicturedimension,bandwidthiseciently pictureeverytseconds. videoconferencingsystemsmustexploitreal-timeencodersthatguaranteethedeliveryofa byconsideringthenatureoftheapplicationgeneratingtrac.accordingtorequirement6, savescheduling,i.e.,thechoiceofthetfstobereservedtoavideoconferencecall,issimplied considerm=1keepinginmindthatthewhatisdescribedinsection3.3.3appliestothe generalcase.ineachtimecycle,fibitsareallocatedduringonetf. siderthatonlyintra-framecodingisperformed.underthisassumption,resourcereservation mustbeanintegermultiplemofthevideoframeperiod;forthesakeofsimplicity,wenow isperformedasdescribedinsection3.3fortransmissionofrawvideo.thetimecycleduration ThebitsencodingapicturearebueredfromwhentheyareproducedtowhentheTF Tobetterunderstandtheimplicationsofschedulingontheend-to-enddelay,werstcon- codingtimecm,eachpicturespendsinthepacketizationfunctionbuertheapplication AsexplainedinSection3.3.2,iftheclockdrivingthecapturecard(andhencethepaceatwhich synchronizationcomponentofthenetworkshapingdelaysas packetizationfunction.temporarilyassumingthatencodingeachpicturetakesthemaximum reservedfortheirtransmissionisscheduled.thisbueringcantakeplace,forexample,inthe 46 n2[;t],asshowninfigure37.

Cheerleaders - Picture ratio 2 Hockey - Picture ratio 2 Average Quantization Parameter 8 6 4 2 Average Quantization Parameter 8 6 4 2 3 6 9 12 15 4 Mb/s 8 Mb/s 16 Mb/s Cheerleaders - Picture ratio 3 3 6 9 12 15 Hockey - Picture ratio 3 Average Quantization Parameter 8 6 4 2 Average Quantization Parameter 8 6 4 2 3 6 9 12 15 3 6 9 12 15 Cheerleaders - Picture ratio 4 Average Quantization Parameter 8 6 4 2 Figure33:AverageQuantizationParameterinControlledMPEGEncoding. 47 Hockey - Picture ratio 4 3 6 9 12 15 Average Quantization Parameter 8 6 4 2 3 6 9 12 15

Bikes - Picture ratio 2 Table Tennis - Picture ratio 2 Average Quantization Parameter 8 6 4 2 Average Quantization Parameter 8 6 4 2 3 6 9 12 15 4 Mb/s 8 Mb/s 16 Mb/s Bikes - Picture ratio 3 3 6 9 12 15 Table Tennis - Picture ratio 3 Average Quantization Parameter 8 6 4 2 Average Quantization Parameter 8 6 4 2 3 6 9 12 15 3 6 9 12 15 Bikes - Picture ratio 4 Average Quantization Parameter 8 6 4 2 Figure34:AverageQuantizationParameterinControlledMPEGEncoding. 48 Table Tennis - Picture ratio 4 3 6 9 12 15 Average Quantization Parameter 8 6 4 2 3 6 9 12 15

5 Cheerleaders - Picture ratio 2 5 Hockey - Picture ratio 2 45 45 SNR (db) 4 35 SNR (db) 4 35 3 3 25 5 3 6 9 12 15 4 Mb/s 8 Mb/s 16 Mb/s Cheerleaders - Picture ratio 3 5 25 3 6 9 12 15 Hockey - Picture ratio 3 45 45 SNR (db) 4 35 SNR (db) 4 35 3 3 25 25 3 6 9 12 15 3 6 9 12 15 SNR (db) 5 45 4 35 3 25 Figure35:SNRinControlledMPEGEncoding. Cheerleaders - Picture ratio 4 5 49 45 4 35 3 25 3 6 9 12 15 SNR (db) Hockey - Picture ratio 4 3 6 9 12 15

5 Bikes - Picture ratio 2 5 Table Tennis - Picture ratio 2 45 45 SNR (db) 4 35 SNR (db) 4 35 3 3 25 5 3 6 9 12 15 4 Mb/s 8 Mb/s 16 Mb/s Bikes - Picture ratio 3 5 25 3 6 9 12 15 Table Tennis - Picture ratio 3 45 45 SNR (db) 4 35 SNR (db) 4 35 3 3 25 25 3 6 9 12 15 3 6 9 12 15 SNR (db) 5 45 4 35 3 25 Figure36:SNRinControlledMPEGEncoding. Bikes - Picture ratio 4 5 5 45 4 35 3 25 3 6 9 12 15 SNR (db) Table Tennis - Picture ratio 4 3 6 9 12 15

Rate encodedpicturesexittheencoder)isnotsynchronizedwiththenetworkinterface,sas T 2T 3T 4T 5T 6T Rate Otherwise,theend-to-enddelayisgivenby variesinthegiveninterval.ifthetwocapturecardandnetworkinterfacearesynchronized, nisconstantandthetimeinwhichpicturesarecapturedcanbechosensothatsas Figure37:I-frameOnlyCodingandBandwidthReservationwithTimeDrivenPriority C S M n nslowly T sentassoonasencoded,itwouldbedelayedbythedecodertocompensatetheprocessing functionlongerthansas IftheencodingtimeofapictureislessthanCM,itisbueredbythepacketization n.thisdoesnotaecttheend-to-enddelaybecauseifthepicturewas f TDP VBR?I=CM+SAS n+ltf+d+pd n=. (16) Cycle delayvariation.thus,thiscongurationallowstosimplifythereceiverthatdoesnotneedto TFsintendedforsendingI-framesandthoseforP-frames.Thetimecyclemustbesettoan forthetwodierenttypesofframes:dierentamountsofbitsshouldbereservedinthe Whenbothintra-frameandpredictivecodingareexploited,twodierentboundsareimposed 4.3.4ComplexScheduling compensatenetworkjitterandprocessingdelayvariation. integermultiplemofthegopperiodandnmtfsmustbereservedwithinthetimecycle. Figure38showsasamplereservationwithM=1andN=4;theupperdiagramshowsthe amountofbitsgeneratedtoencodeeachpictureassumingthattheyareinstantlyavailable 51

Rate afterthemaximumencodingdelaycmfromthecapture1.thelowerdiagramshowsthe Figure38:TimeDrivenPriorityandComplexScheduling. T 2T 3T 4T 5T 6T Time Rate C AS delayisgivenby(16)asinthecaseinwhichonlyintra-framecodingisexploited.nevertheless, choiceofthetfstobereservedoneachlinkonthepathbetweensenderandreceiveriscalled amountofbitsreservedinthetfs:fiinonetfandfpinthefollowingn?1tfs.the M Sn Time T dierentvariationinterval;form=1,sas ofthevideoconferencecall. inthiscasetheapplicationsynchronizationcomponentofthenetworkshapingdelayhasa complexscheduling.thechoiceofthetfsimpactsbothnetworkperformance(intermsof maximumnumberofreal-timeconnectionsconcurrentlysupported)andtheend-to-enddelay AssuminguniformdistributionwithinthetimecycleofthereservedTFs,theend-to-end f Cycle where,duetotheslowmotioninthevideosequence,itisconvenienttousealonggop(large ofthegopperiodisdenitelynotacceptableespeciallyinvideoconferencingapplications N).Thus,synchronizationbetweenvideoencoderandnetworkinterfaceisessentialtokeep theend-to-enddelaybelowthe1msboundandthebuerofthepacketizationfunction small. n2[;nt].anetworkshapingdelayoftheorder alimitofthesystem,butinsteadallowsthedecodertobesimplied. Assumption3Thecapturecard(aswellastheencoder)issynchronizedwiththenetwork interface. 1Ithasbeenalreadyshowninthecaseofintra-frameonlycodingthatthisassumptiondoesnotconstitute IftheTFsreservedtoavideoconferencecallarenotuniformlydistributedwithinthetime 52

Bits I frames Bound P frames Bound cycletheinterarrivaltimebetweenvideoframesisnotconstant.thereceivercompensateto thisvariabilitybybueringvideoframesreceivedbeforescheduleanddelayingthewholevideo T 2T 3T 4T 5T 6T Time stream.sincethisdelayisduetotheadaptationofthetransmissiontimetothecharacteristics Rate ofthenetwork,wecategorizethisdelayasaschedulingcomponentofthenetworkshaping delaysschedfigure39:timedrivenpriorityandcomplexscheduling. AS S wheressched byawiseschedulinginthenetwork. n.ifassumption3holds,theend-to-enddelaycanbewrittenas TDP?CxSc n VBR Time T ThedecodingtimeDcanbereducedifthedecoderdoesnothavetowaitthewholepicture 4.3.5ReducingDecodingTime n2[;nt]:itisxedwhenthevideoconferencecallisplacedandiskeptsmall f =CM+SSched n+ltf+d+pd ofthem(figure39),thantoallocateasingletfduringwhichawholeencodedpictureis beforestartingprocessingit.thiscanbeobtainedbyinsertingencodedvideoframesin sent(figure38).choosingthistfsencompassestwonontrivialissues:determiningtheir smallerpacketsthataresentassoonasthecorrespondingbitshavebeenproducedbythe outlined,butasolutionisnotproposedbecauseitisoutofthescopeofthiswork. Inthefollowingtheseproblemsaredescribedandthetrade-osbetweendierentchoicesare positioninsidethetimecycleandtheirnumbersothattheend-to-enddelaybeminimized. encoder.thisdoesnotaectanyothercomponentoftheend-to-enddelaygivenby(16). ItisthusbettertoallocatemorethanoneTFperpictureandsendapacketineach 53

forthepictureoverallthetfs,theremainingreservedtfswillnothaveenoughcapacity tocarrytheotherbitsthathastobesentinthescheduledtf.ontheotherhand,ifthe bitsactuallyproducedtoencodethepictureisveryclosetotheglobalamountofbitsreserved underutilized(itcanbeexploitedbybesteorttrac),butbecause,ifthetotalamountof bytheencoder,otherwisetheamountoftransmitteddataissmallerthantheonereserved. Thisiscalledencoderunderowanditcriticalnotjustbecausetheallocatedbandwidthis WhenareservedTFisscheduledonthesenderlink,enoughbitsmusthavebeenproduced TFsreservedforapicturearechosenlater(withrespecttothebeginningofthevideoframe period),thebenetofusingmoretfstotransmitapictureisreduced,i.e.,theend-to-end delayisincreased. imagecomplexityandmotion,thanthescenesusedintheseexperiments,thusshowingamore andfigure41whenencodingframeswithboundeddimension.thecurvesshowthatthe productionrateisquiteconstant,i.e.,thepositionoftfscanbeestimatedwithasimple linearmodelwhichintroducessomesafetymargin(byshiftingthetfstowardstheendof probabilityofencoderunderow.moreover,avideoconferencescenehasusuallymoreuniform thevideoframeperiod).figures4and41showthatasafetymarginoffewmsyieldslow ThesoftwareencoderthathasbeenusedinthisworkshowsthetimingplottedinFigure4 regularproductionrate. bythemodel,thebitsthatcannotbetinthetfsreservedforthepicturecanbesentas besteort;ifthenetworkiscongested,theydonottimelygettothereceiverandthevisual thehigherthenumberofreservedtfs,thehighertheprobabilityofencoderunderow(the qualityofthereconstructedscenetemporarilydegrades. thesmallerthetransmissionunit,theearlierthedecodercanstartdecoding.nevertheless, Iftheinstantaneousamountofbitsproducedbytheencoderisbelowtheamountpredicted thedimensionofthetransmissionunit. modelaveragestherealproductionrateonashorterinterval).moreover,eventhoughinthis workpacketheaderoverheadisbeingneglected,ithastobetakenintoaccountwhenchoosing ThenumberofthereservedTFsperpictureshouldbekeptashighaspossiblebecause 4.4AsynchronousPacketSwitching Avideoconferencingsystemcanbeconstructedoveranasynchronouspacketswitchednetwork. andthetransmissionoverhead,ontheotherhand. thereductioninthedecodingdelay,ononehand,andthepossibilityofencoderunderow TheamountofbitstobereservedineachTFmustbechosenndingatrade-obetween AssoonastheencoderproducesenoughbitstoassembleapacketofdimensionPs,thepacket issentintothenetworkwhereitisexperienceavariablequeueingdelay.asshowninfigure42, thereceivermustexploitareplaybuertocompensatethenetworkdelayvariation.theend toenddelayisthusgivenby 54

Number of bits Number of bits Number of bits 3 25 2 15 1 5 3 25 2 15 1 5 3 25 2 15 1 5 Cheerleaders 5 1 15 2 25 3 Video frame priod Cheerleaders 3 35 4 45 5 55 6 Video frame priod Cheerleaders 6 65 7 75 8 85 9 Video frame priod Cheerleaders 3 25 2 15 Figure4:ProductionofBitswhenEncodingthe\Cheerleaders"Sequence. 1 5 55 9 95 1 15 11 115 12 Video frame priod 3 25 2 15 1 5 12 125 13 135 14 145 15 Video frame priod Number of bits Number of bits

Number of bits Number of bits Number of bits 3 25 2 15 1 5 3 25 2 15 1 5 3 25 2 15 1 5 Hockey 5 1 15 2 25 3 Video frame priod Hockey 3 35 4 45 5 55 6 Video frame priod Hockey 6 65 7 75 8 85 9 Video frame priod Hockey Number of bits Number of bits 3 25 2 15 Figure41:ProductionofBitswhenEncodingthe\Hockey"Sequence. 1 5 56 9 95 1 15 11 115 12 Video frame priod 3 25 2 15 1 5 12 125 13 135 14 145 15 Video frame priod

SenderandReceiver.Async Figure42:VideoconferencingSystemCongurationwithPacketSwitchedNetworkbetween wherecmisthemaximumtimerequiredtoencodeapicture,qmisthemaximumqueueing Disthedecodingdelay.Usingsmallpacketsandadecoderwhichstartsdecodingvideoframes Frame Packetization Packetization assoonasdataarereceived,canreduced.anyway,theend-to-enddelayisdominatedby delay,er2[;q]istheexcessresynchronizationdelayintroducedbythereplaybuer,and VBR=CM+Ps C+P+QM+Er+D+Pd Grabber Encoder Decoder Adaptor boundariesofthenetwork,eventhoughitintroducesanetworkshapingdelay.resources insection3.4.2,resourcereservationismoreecientiftracshapingisperformedatthe Resourcesarereservedinthenetworkinordertoboundthequeueingdelay.Asdiscussed 4.4.1TracShapingatNetworkBoundaries CMandQM. thedescriptiongivenatresourcereservationtime. averagerate,correspondingtotheshapedtrac;thenetworkguaranteesthequalityonthe arereservedinthenetworkbasedonthetracdescription,givenintermsofburstinessand networkshapingdelaysts characteristicsoftheshapedtrac.thereceiverhastocompensateitbymeansofareplay service(i.e.,theboundqmonthequeueingdelay)onlyiftheactualtraciscompliantwith buerwhichintroducesanetworkresynchronizationdelay.thus,eachpacketexperiencesa naturalbitgenerationrateoftheencoder,theimplementationofthetracshaper,andthe Thedelaygloballyexperiencedbyapictureduetothetracshaperdependsonthe caseofvbrmpegencodedvideothesametaskisharderanditisnotthegoalofthiswork, wedonotanalyzests isgivenby andtheimpactontheend-to-enddelayinthecaseoftransmissionofrawvideo.sinceinthe Section3.4.2,wehavediscussedtheconstraintsonthechoiceoftheleakybucketparameters ninmoredetail.theend-to-enddelayofthevideoconferencingsystem npartlyinthetracshaperandpartlyinthereplaybuer.in Async?TS VBR=CM+STS n+ps 57 C+P+QM+Er+D+Pd

4.4.2AdaptingtheEncodedVideoStreamtotheNetwork andtokenpoolsizeadeterminetheburstinessandaveragerateoftheshapedtrac.ifthe SincethetracpatterngeneratedbyanaturalVBRMPEGencoderisnotknowninadvance, discardedbyeitherthetracshaperitself,oratracpolicingfunctioninsidethenetwork[21]. itcanbeincompatiblewiththeshapedtracdescription.thenoncompliantpacketscanbe Forexample,ifaleakybucketisexploitedtoshapethetrac,thetokengenerationrateB whenthegopislarge.eventhoughtechniqueshavebeenproposedtolimittheeectof theexcesstracmustbeeitherdiscardedorsentinthenetworkwith"besteort"service. characteristicsoftheencodedvideoarenotcompatiblewiththevalueschosenforbanda Eventhoughabuerisinsertedbeforetheleakybuckettoadaptthevideostreamtothe loss[6],itshouldbebetterforthevideoconferencingsystemtoavoidlossinordertodeliver tracdescription,itcanoverowifthetwoaretoodierent. thehighestpossiblequality. ThelossofpacketsisnotacceptableinthetransmissionofMPEGencodedvideo,especially tracdescription. controlfunctiontunestheparametersofthebasicmpegencoderaccordingtothetrac MPEGencoders,thisapproachdoesnotguaranteeagainstpacketsnotcompliantwiththe sendsasbesteorttracpacketsthatcannotbeadaptedtothetracdescription.arate descriptionusedtodrivethetracshaper[24,23].duetotheunpredictableoutputof Alternatively,theratecontrolfunctioncantunetheMPEGencoderparametersbasedon TheMPEGencodingprocesscanbecontrolledtoavoidthatthetracshaperdiscardsor leakybucket)[2,4],asshowninfigure43.ifthisrequirestosignicantlydegradethevisual feedbackinformationreceivedfromthetracshaper(e.g.,thefullnessofthebuerprecedinga qualityofpictures,theresourceallocationisrenegotiatedaccordingtoarate-quantization modelthatistunedastheencodingprogresses. reconstructedonthereceiverwithacceptablequality.theencodedstreamcanbepartitioned twoseparatelayers,thehighprioritylayer-whichcontainsthemostcriticaldataaccording complytothetracdescriptionbecauseitneedsserviceguarantees,whilethelatterissent streamtotheserviceprovidedbythenetwork.thebitsencodingeachframearedividedinto tothesensitivityofthehumanvisualsystem-andthelowprioritylayer.theformermust asbesteort.eventhoughthelowprioritycomponentgetslost,thevideosequencecanbe Hierarchicalorlayeredencoding[19]providesanothermeanstoadapttheencodedvideo correspondingstreambettertothetracdescription. dynamicallyaccordingtofeedbackinformationfromthetracshaper(e.g.,thefullnessof thetokenbucket)[18].thefractionofdatasentatthehighprioritylayertobettertthe availablebandwidthwhenthenetworkisnotoverloaded,andgettingreasonablevisualquality whenitiscongested.onthecountrary,theapproachbasedoncontrollingtherateofthe MPEGencodervariesthequalityofimageswiththeobjectiveofttingtheencodedvideo Hierarchicalencodingcangetthebestfromapacketswitchednetworkbyusingallthe 58

B A Target Base Rate Encoder streamputsaharmfulburdenonthenetworkwhenitisoverloaded.duringcongestion, Figure43:MPEGEncoderControlledusingFeedbackfromaTracShaper. Quantization Rate nation.evenworse,lowprioritypacketswasteresourcesuntiltheyarediscarded.thewaste Parameter isevenlargerifpacketsmaketheirwaytothedestinationandtheretheyarediscardeddue thoughthenetworkcouldpossiblysupportahighrate(i.e.,higherquality). streamintothetracdescription.thus,visulalqualitycansometimesbedecreasedeven sendinglowprioritypaketsdoesnotyieldanyadvantagesincetheydonotgettothedesti- toexcessivedelay11. Nevertheless,thehierarchicalencodingapproachisnotecientbecausethelowpriority Model of a System Target Decoder Buffer AcircuitswitchednetworkcanbeusedforthetransmissionofVBRMPEGencodedvideo. 4.5CircuitSwitching whereswistheswitchingdelayandpthepropagationdelayinthenetwork.thisexpression issimilarto(11)giveninsection4.2fortheend-to-enddelayofavideoconferencingsystem Ifthevideoconferencecallisallocatedacircuitwithbandwidthlargerthantheinstantaneous rateoftheencoderandtheend-to-enddelayis thecapacitycoflinksisdedicatedtothevideoconferencecall. thatusesvbrmpegencodingandadirectlinkbetweensenderandreceiver.whena circuitswitchednetworkisexploitedtheswitchingdelayhastobeadded;moreover,notall CS VBR=CM+Sw+P+D+Pd; 11Takeintoaccountthatthedelayexperiencedbybesteorttrachasalargeboundandhighvariability. 59

showninfigure44.itsmoothestherateoftheencodertotitinthecircuitbandwidthatthe Figure44:VideoconferencingSystemTransmittingVBRMPEGvideooveraCircuitSwitched expensesofanetworkshapingdelayscs belowthepeakrateofthempegstreamifabuerisusedbetweenencoderandnetwork,as Network. Section4.4.1aboutthenetworkshapingdelaySTS encoder,alargepartofitsbandwidthiswasted.thebandwidthofthecircuitcanbereduced packetswitching,appliesalsotoscs SincethecircuithasbeendimensionedtosupporttheburstinessontheVBRMPEG Frame Network Interface andthecircuitswitchednetworkisequivalenttotheoneduetoatracshaperwithaverage n.infact,theshapingperformedbythesmoothingbuer naddedtotheend-to-enddelay.whatisdiscussedin nintroducedbyatracshaperwhenusing Grabber Encoder Network Interface Decoder Adaptor bandwidthbandmaximumburstiness1bit(e.g.,aleakybucketwithtokengenerationrate totheend-to-enddelaywhichisgivenby receiversidecompensatesthevariationofthistimethusyieldingthexedcontributionscs ontherateofthempegencoderandthebandwidthofthecircuit.areplaybueronthe BandtokenpoolsizeA=1).Picturesarebueredinthesenderforatimethatdepends aleakybucket,whichhasbeendescribedinsection3.4.2;whenitoverowspartoftheencoded streamisdiscarded.dimensioningthebuerandthecircuitbandwidthsothatdiscarding Thebuerbetweenencoderandnetworkhasthesamepurposeasthebuerattheingressof CS?TS VBR=CM+SCS n+p+sw+d+pd n ofpacketsisavoidedisnotpossiblebecausethebitrateproducedbythempegencoderis notknowninadvance.thus,thebitgenerationrateoftheencodermustbemonitoredand regulatedbyaratecontrolfunctionwhichexploitsthebuerstatusasfeedbackinformation. isactuallyacbrmpegencoderwhichisthesubjectofthenextsection. ThesystemcomposedbythebasicMPEGencoder,thebuer,andtheratecontrolfunction 5TransmissionofCBRMPEGVideo ThebasicMPEGcodingprocessproducesavariablebitratestream.Ashasbeenshownin theprevioussection,topreventloss,thenaturalvariabilityofthebitgenerationratemustbe controlledandadaptedtothetransmissionservice.ifthetransmissionservicerequiresdata tobesentatconstantbitrate(cbr),acbrmpegencoderisthechoiceforbuildingthe videoconferencingsystem. 6

Figure45:BasicMPEGEncoderandEncoderBuer. Rate Rate B Encoder Buffer alsopredictivecodingareconsiderednext.lastly,thetransmissionwithdierentschemesis viewofdelaywhenonlyintra-framecodingisused;theimplicationsonthedelaywhenusing thatinuencethem.first,thempegencoding/decodingsystemisanalyzedfromthepointof Inthefollowing,thetimeneededtoencodeandtransmitCBRMPEGvideoisstudiedfor furtherstepswiththeaimofhighlightingthedierentcomponentsandthesystemparameters transmissionschemeadoptedandwidelyvariesaccordingtotheparametersoftheencoder. ACBRMPEGencoder/decodersystemintroducesadelaythatisindependentofthe Time Time analyzedinordertodevisetheend-to-enddelayinvariouscongurations. Base Encoder iscomposedonlyofi-frames)isrstconsidered;basically,thisisthecompressionprocess Thecaseinwhichonlyspatialredundancyiseliminated(i.e.,theencodedvideosequence 5.1Intra-frameCodingOnly ACBRMPEGencoderisbuiltbysmoothingthenaturalrateofabasicMPEGencoderinto aconstanttargetrateb.asshowninfigure45,thesmoothingisperformedbytheencoder 5.1.1CodingShapingDelay alsoapplytoavideoconferencingsystemwhichexploitjpegtocompressvideoframes. recommendedbymovingjpeg[7][25]andthusthebasicprinciplesdiscussedinthissection bueredinthedecoderbueruntilawholepictureisreceivedandcanbedecode.apicture thatvideoframesarecontinuouslydisplayedataxedrate.thecompositionoftheprocessing buerdelayseachbitdependingonthenaturalbitrateandthetargetbitrate.also,bitsare ThisistheprocessingdelayoftheCBRencoderthatdependsonthepicturedimension.Since itnotconstant,thedecodermustcompensateittocomplywithrequirement3whichimpose encodedwithfbitstakesatimef=btoexittheencoderbuerandenterthedecoderone. buer,whichislledatthenaturalbitrateandemptiedatthetargetbitrateb.theencoder delayoftheencoder/decodersystemandtheprocessingresynchronizationdelayintroduced bythedecoderisaconstantcodingshapingdelaysc. 61

Natural Buffer Target Base wheremaxseqfisthemaximumnumberofbitsusedtoencodeapictureovertheallsequence. Figure46:BasicEncoderandRateControlFunction. (17)guaranteesthatinthetimebetweenwhenapictureisgrabbed,towhenitisscheduled Quantization fordisplaying,uptomaxseqfbitscanexittheencoderbueratrateb.inorderforthe SincebitsexittheencoderbueratrateB, Parameter encodedvideostreamtobecontinuous(i.e.,actuallyhavingaconstantrate), ScmaxseqF B; Model a System Target Decoder Buffer areproduced.(18)showsthattheend-to-enddelayofavideoconferencingsystemexploiting otherwisetherewouldbeatimeintervalbetweentwosubsequentframesduringwhichnobits acbrmpegencoderislargerthanthevideoframeperiod. itisanalogoustothenetworkshapingdelaysts toavbrmpegencoder(seesection4.4.1).ifabasicmpegencoderfreelyllsanite dimensionbuerwhichisemptiedaccordingtoaxedtracdescription,thebuercan Eventhoughthecodingshapingdelayisnotrelatedtothetransmissionoveranetwork, undergoestoaverystricttracdescription(beingitconstant).thus,aratecontrolfunction possiblyoverow.insection4.4.2wesurveyedsomeproposalstocontrolthebasicmpeg todiscardpackets.inthecaseofacbrencoder,therateatwhichthebuerisemptied encodingbasedonfeedforwardorfeedbackinformation,inordertoavoidatracshaper nintroducedbyatracshapercascaded mustbeintroducedtocontrolthebasicmpegencoderinordertoavoidtheencoderbuer tochangethenaturalbitrateandavoidthebuertounderoworoverow. Figure46showsthecomponentsofaCBRMPEGencoder.Inprincipletheratecontrol functionmonitorsthefullnessoftheencoderbuerandadjuststhequantizationparameter tooveroworunderow. 5.1.2RateControlFunction 62 ScT; (18)

asystemtargetdecoderwiththestreamdoesnotoveroworunderowitsbuer,whose dimensionisincludedinthempegstream[11].thempegstandardspeciesthatthe systemtargetdecoderisfedattheconstantrateindicatedinthestreamanditretrievesdata fromitsbueroneaccessunitatatime.theaccessunitisawholepictureandallthebits timespeciedinthempegstream.thebuerofthesystemtargetdecoderunderowsifa encodingavideoframeareremovedwhenitsdecodingisstartedaccordingtothedecoding ThecomplianceofanencodedvideostreamwiththeMPEGstandardrequiresthatfeeding wholepictureisnotpresentinthebuerwhenitsdecodingisscheduled. timetoencodeapictureisassumedtobenull,thefullnesslevelsofthetwobuersarestrictly updatedmodelofthebuerofasystemtargetdecoder,asshowninfigure46.itislledat sothatthestreamcomplianceisgranted. asshowninfigure47.thequantizationparameter(andthusthenaturalbitrate)istuned thetargetrateb,andemptiedbyonepictureattheinstantinwhichdecodingisscheduled, Actually,iftheencoderbuerhasthesamesizeofthesystemtargetdecoderoneandthe Thus,besidemonitoringtheencoderbuer,theratecontrolfunctionshouldkeepan tothedimensionofpicturesandthetargetrateb.ifthevirtualbuerdoesnotunder- MPEGstandardprovidesguidelinesonhowtomanagethevirtualbuerandmodifythe oworunderows,neithertheencoderbuernorthesystemtargetdecoderbuerdo.the decoderoneunderows(overows). related.inparticulartheencoderbueroverows(underows)ifandonlyifthesystemtarget quantizationparameter[9];anyway,eachcbrmpegencoderimplementationcanuseits ownapproach[28,26,5,23]. Inpractice,theratecontrolfunctioncanmodelvirtualbuerwhichisupdatedaccording thesoftwareencoderdvdencwiththeobjectiveofdevisingitsimpactontheend-to-enddelay. accordingto(17)themaximumnumberofbitsusedtoencodeapictureprovidesalowerbound thedimensionofthesystemtargetdecoderbuerandisdrainedattheconstantbitrate onthecodingshapingdelay,weanalyzeinmoredetailthevirtualbuerimplementationof B.Beforeencodingapicture,theratecontrolfunctionsetsthemaximumandminimum Thisthevirtualbuerconstrainsthenumberofbitsusedtoencodeeachpicture.Since thatthebuerisnotemptiedbeforetheendofthevideoframeperiod.thevbvisthen updatedaccordingtotheactualnumberofbitsproduced.inthefollowingsection,thelower dimensionsofthepicture.themaximumdimensionissetbyconservativelyassumingthat itmustnotexceedthespaceavailableinthebuer12.theminimumdimensionguarantees Theratecontrolfunctionmodelsabuer,calledVideoBuerVerier(VBV),thathas wouldnotoverow. andupperboundsonthedimensionofpicturesinancbrmpegstreamaregiveninterms ofthesizeofthevbv. 12ThisconformstoconservativeassumptionthatevenifallthebitswereinjectedintheVBVinstantly,it 63

Figure47:StandardDecoderBuerModelKeptbytheControlFunction. Rate Rate Rate ofimageswouldnotbeuniform.insteadpicturedimensionisallowedtovaryandiscontrolled B accordingtothevbv. encodeeachpicture.eventhoughthiswouldguaranteeaconstantbitrate,thevisualquality InthevideoframeperiodT,aCBRMPEGencoderworkingatthetargetrateBproduces BTbits.TheratecontrolfunctioncanbedesignedinordertouseexactlyBTbitsto 5.1.3DimensionofEncodedPictures Time Encoder Buffer Time Buffer T Time Claim1ThedimensionoftheVBVistheupperboundonthedimensionofpicturesinan Base Encoder wherevsisthedimensionofthevbvandfisthedimensionofapicture. MPEGcompliantstream: ProofTheVBVmustbelargerthanthelargestencodedpicturebecauseitmustcontaina wholepicturebeforeitisstartedbeingdecoded.moreover,asthevbvoverowmust beavoidedtoguaranteecompliancewiththempegstandard,theratecontrolfunction Vsmax seqf showhowthislowerboundisdetermined,werstanalyzehowpicturedimensionaectsthe thisisnotarestrictionbecauseifthevbvdoesnotunderow,theencoderbuernever encoderbuerlevel.weassumethattheencoderbuerandthevbvasthesamedimension; ThelowerboundonencodedpicturedimensionisalsorelatedtotheVBVdimension.To controlsthebasicncodersothatthenumberofbitsusedtoencodeapictureisnotarger containsmorebitsthanthevbvdimension. thanthevbvdimension. BTbits.Ifsomebacklogispresentinthebueratthebeginningofthevideoframeperiod, ratebduringthetimetbetweentwopictures,apictureshouldbeencodedwithatleast thenumberofbitsusedforencodingthepicturecanbesmaller.thus,thelowerboundfm Inordertohaveenoughbitsintheencoderbuersothatitisemptiedattheconstant 64

Bits T 2T 3T Time largestbacklog. exempliedinfigure48.theupperdiagramshowsthattheencoderprocessesthebitsofeach rawimageduringasinglevideoframeperiod(thindashedline).thediagramalsoplotsthe Figure48:BitProductionRateandEncoderBuerFullness. T 2T 3T Time line);thetotalnumberofbitsperpicturevariesfordierentpictures.thecbrencoder onpicturedimensionistheminimumallowedpicturedimensionwhenthebuercontainsthe numberofbitsgeneratedbythebasicencoderversustimeforeachpicture(thickcontinuous Processed emitsbitsatconstantrateb(asshownbythethickdashedlineinthelowerdiagram),thus Thebacklogisduetobitsencodingthepreviouspicturethathasnotbeensentoutyet,as Natural Target sendingoutthesameamountofbits(bt)ineachvideoframeperiod(lastpointofthe Buffer Level thickdashedlineintheupperdiagram).ifthedimensionofapictureexceedsbt(e.g.,the rstpictureinfigure48),excessbitsaretemporarilybueredintheencoderbuer(whose levelisdepictedbythethincontinuousline)untiltheyhavebeenallsentout(aftertheend ofthevideoframeperiod). encodingapicturetheratecontrolfunctionchecksthevbvandallowsthepicturetobelarge enoughtollit.ifthemaximumallowedamountofbitsisproduced,thelevelofthevbv (andtheencoderbuer)attheendofthevideoframeperiodis ThelargestbacklogVbremainsinthebuerafterapicturehaslledit.Beforestarting Vb=Vs?BT 65

becauseduringthevideoframeperiodbtbitshavebeendrainedfromthebuer.the numberofbitsthatmustbeinsertedinthevbvsothatitcontainsatleastbtbits,i.e., ratecontrolfunctiondeterminestheminimumalloweddimensionofthenextpictureasthe theamountofbitstobedrainedduringthevideoframeperiod: Thisisthelowerboundonthepicturedimension. apicture.moreover,theminimumnumberofbitsusabletoencodeapictureisbounded bythevbvdimension,thevideoframerate,andthetargetrateoftheencoder.sincethe naturaldimensionofmpegencodedpicturesishighlyvariable(seesection4.1.3,themore Insummary,thedimensionoftheVBVisthemaximumnumberofbitsusabletoencode Fm=BT?Vb=2BT?Vs thedimensionofpicturesisallowedtovary,themoreuniformistheirvisualquality.the maximumpossiblevariationofpicturedimensionisgivenby isfundamentaltokeepthevisualqualityuniformthroughoutpicturesandscenes.thetask ismademoredicultbythefactthat,underassumption3ofcontinuousplayingmadein function.theratecontrolfunctionmustusewiselytheallowedvariabilityinpicturedimension pictures. ThelargertheVBVdimension,thelargerthepossiblevariationofdimensionofencoded Theactualnumberofbitsusedtoencodeeachpictureisdeterminedbytheratecontrol FM=Vs?Fm=2(Vs?BT) (19) picturemustbesmallerthanbt. Claim2Ifapicturehasitsmaximumalloweddimension(i.e.,itllstheVBV),thefollowing ProofWhenapictureisencodedwiththemaximumallowednumberofbits,attheendof intheprecedingones.thisisstatedbythefollowingclaim. Section2.2,theamountofbitsusedtoencodeapictureisconstrainedbytheamountsused 5.1.4StartupShapingDelay thevideoframeperiodinwhichithasbeencaptured,bthasbeendrainedfromthe VBV.BeingthisthenumberofbitsneededtolltheVBV,itprovidesthemaximum Sectionweshowthat,evenifthedimensionofapictureislargerthantheminimumallowed TheratecontrolfunctionusestheVBVtoguaranteethattheMPEGstreamcomplieswith thempegstandardandthattheencoderbuerdoesnotoveroworunderow.inthis alloweddimensionforthefollowingpicture. bythevbv,theencoderbuercanunderow.wealsodescribehowthecbrencodercopes withthissituation. 66

Bits theupperdiagramshowstheamountofbitsprocessedversusthetime.thisislargerthan theamountofbitsproducedbythebasicencoder(continuousline)becauseoftheyielded compression;thepunctualslopeofthelineistheinstantaneousnaturalrate.theobjective ofthecbrencoderistoproducebitsattheconstanttargetrateb,asshownbythethick Figure49:I-framesOnlyEncoding. dashedline;theencoderbuerisusedtosmooththevariationsofthenaturalbitrate,as Figure48showsthetimingassociatedwithintra-framecoding.Thethindashedlinein T 2T 3T Time exempliedbythelowerdiagramoffigure48. Processed Natural Rate levellowers,asshown,forexample,intheupperdiagramoffigure48duringthesecondvideo Whentheinstantaneousnaturalbitrateissmallerthanthetargetrate,theencoderbuer Target Rate situationdepictedinfigure49:atthebeginningoftheencodingofascene,thenaturalrate islowerthanthetargetrate.thiscanbepreventedbywaitingastartupshapingdelaywhen theencodingofascenestarts,beforestartingsendingbitsoutoftheencoderbuer.the whenthebuerisempty,theencoderbuerunderows.thishappens,forexample,inthe frameperiodbythethincontinuousline.ifthenaturalrateissmallerthanthetargetrate startupshapingdelayisobtainedbyxingafullnessthresholdinthebuerandwaitingitto becrossedbeforethempegstreamstartsowingoutofthecbrencoder. targetrate.sincethenaturalrateiscontrolledbytheratecontrolfunction,thethreshold canbeaslowasdesired,providedthattheratecontrolisaggressiveenoughtoincrementthe smallertheareaofthepictureoverwhichthenaturalproductionratecanbesmallerthan visualqualityofimagesisnotuniformbecausethequantizationparameterchangessuddenly. naturalratesothatthebuerdoesnotunderows.nevertheless,thelowerthethreshold,the thetargetrate,i.e.,theareaoverwhichtheratecontrolfunctionmustintervene.thus,the Thethresholdrepresentsthetoleranceofthesystemtoanaturalbitratelowerthanthe Ontheotherhand,thehigherthethreshold,thelargerthestartupshapingdelay. quality(andaggressivecontrolfunction)islefttothespecicencoderimplementation.the Thechoiceofthetradeobetweenlongstartupshapingdelayandnon-uniformvisual 67

Bits chosenvalueofthestartupshapingdelayisnotverycriticalbecause,iftheencoderispowerful Thismeansthatthemainaimoftheencoderbuerisnotreallysmoothingthevariationof enoughthetargetrateismuchlowerthantheaveragenaturalrate,asshowninfigure5. Figure5:FastI-frameOnlyEncoding. T 2T 3T Time Processed Natural Rate 5.1.5ShapingDelayImplementation delaytocopewithactualbitproductionratevariation,i.e.,thefractionofbuertobelled beforestartingtransmittingshouldnotbelargebecausethebuerisemptiedmuchslower thenaturalrate,butloweringit.thus,itisnotnecessarytointroducealargestartupshaping thanitislledontheaverage. Target Rate Afterhavingdescribedwhicharetheboundsonthedimensionofpictures,weanalyzetheir impactonthedelayandhowitisintroducedbythecbrencoder/decodersystem. partlyinthedecoderone.nevertheless,thevalueofthecodingshapingdelayischosenby bueruntilthebuerlevelcrossesapredenedthreshold.inordertokeepthestartupshaping crossedwithinapredenedamountoftime. delaysmall,theratecontrolfunctionmustincreasethenaturalrateifthethresholdisnot introducesthestartupshapingdelaywhenitstartsencodingascene:nobitexitsfromits Thecodingshapingdelayisexperiencedbypicturespartlyintheencoderbuerand Picturesexperiencethestartupshapingdelayentirelyintheencoderbuer.Theencoder stream[11].thisdeterminestheamountoftimespentbyeachpictureinthedecoderbuer sothattheoveralldelaybesc. theencoderthataccordinglysetsthedecodingandpresentationtimestampsinthempeg CBRencoderisdeterminedbytheVBVdimension,asstatedbyClaim1.Asaconsequence, resultingstream.infact,accordingtoequation(17),giventhecodingshapingdelay,picture delay,i.e.,setthetimestamps;nevertheless,thischoiceiscrucialtothevisualqualityofthe dimensioncannotexceedscb.theupperboundonthedimensionofpicturesencodedbya TheMPEGstandarddoesnotstatehowtheencodershouldchoosethecodingshaping 68

to oncethecodingshapingdelayhasbeenset,theencodermustsetthevbvdimensionaccording theend-to-enddelayofthesystem,butdeliversnonconstantquality.theimplementationof ity,andhencevisualqualityoftheencodedstream.thus,asmallcodingshapingdelayreduces InSection5.1.3ithasbeenshownthatVBVdimensionaectspicturedimension,itsvariabil- theratecontrolfunctionmustndatradeo. accordingly),theproductionofasuitablempegstreamisguaranteedbytheratecontrol Oncethestartupandcodingshapingdelayshasbeenchosen(andtheVBVdimensionset VsScB (2) possibletodecreasethebitproductionrate,atworstbystoppingencodingandusinga particularcodeinthempegstreamthatrepresentsambinwhichallthedctcoecients naturalbitratebecauseunencodedpicturesaremuchlargerthanencodedones.itisalways functionavoidingthevbvtooveroworunderow.itisalwayspossibletoincreasethe arenull. 5.1.6ExperimentalData ThedvdencsoftwareMPEGencoderusedtoprovidetheexperimentaldatashowninthis disks.thus,itisnotoptimallydesignedforareal-timevideoconferencingsystem.asa consequence,thecontrolfunctionofthecbrencoderisnotaggressiveenoughtoproperly workhasbeenimplementedformpegcompressionofvideosequencestobestoredonvideo function,thedimensionofthebuermustbechosenquitelargerthatthisminimumandthus thetargetrateforawholevideoframeperiod.duetothelimitedaggressivenessthecontrol limitpicturedimensionwhenthevbvissmall. deliverpicturessmallerthanbt;inthiscasetheencoderwouldnotbeabletoemitbitsat itsimpactonpicturedimensionisnotclearlyvisible. TheminimumVBVdimensionisBT.AccordingtoClaim1,asmallerdimensionwould (averagedonthewholeframe)decreases,asshowninfigure52.accordingly,thequalityof congurationsdeliverthesamecodingshapingdelay. throughoutthiswork.threedierenttargetratesareusedfortheencodingofeachsequence variation.theratiobetweenvbvdimensionandtargetrateiskeptconstantsothatallthe inordertoshowhowtheparameterinuencesthenumberofbitsperframeproducedandits Figure51showsthenumberofbitsproducedbyintra-framecodingthefoursequencesused theencodedsequencebecomeshigher,asshownbythesnrplottedinfigure53. accordingtoequation(19),morevariabilityisallowedforpicturedimensionasthetarget rateincreases.thisisconrmedbytheplotsinfigure51. Astherateincreasesthedimensionofframesgrowslargerandthequantizationparameter SinceintheexperimentstheratiobetweenVBVdimensionandtargetrateisconstant, 69

Cheerleaders Hockey 7 7 Picture dimension (Kb) 6 5 4 3 2 Picture dimension (Kb) 6 5 4 3 2 Picture dimension (Kb) 1 7 6 5 4 3 2 1 1 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 192 kb 8 Mb/s - Buff 384 kb Bikes 16 Mb/s - Buff 768 kb Table Tennis Figure51:FrameDimensioninIntra-frameOnlyCBREncoding. 7 6 5 4 3 2 7 1 3 6 9 12 15 3 6 9 12 15 Picture dimension (Kb)

Average Quantization Parameter Average Quantization Parameter 1 9 8 7 6 5 4 3 2 1 Cheerleaders Average Quantization Parameter 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 192 kb 8 Mb/s - Buff 384 kb Bikes 16 Mb/s - Buff 768 kb Table Tennis 1 1 Figure52:AverageQuantizationParameterinIntra-frameOnlyCBREncoding. 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 1 71 1 3 6 9 12 15 3 6 9 12 15 Average Quantization Parameter 1 9 8 7 6 5 4 3 2 1 Hockey

5 Cheerleaders 5 Hockey 45 45 SNR (db) 4 35 3 SNR (db) 4 35 3 25 25 SNR (db) 2 5 45 4 35 3 25 2 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 192 kb 8 Mb/s - Buff 384 kb Bikes 16 Mb/s - Buff 768 kb Table Tennis 5 Figure53:SNRinIntra-frameOnlyCBREncoding. 45 4 35 3 25 72 2 3 6 9 12 15 3 6 9 12 15 SNR (db) 2

VBV fullness (%) VBV fullness (%) 1 9 8 7 6 5 4 3 2 1 1 9 8 7 6 5 4 3 2 1 Cheerleaders VBV fullness (%) 1 9 8 7 6 5 4 3 2 1 Hockey 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 192 kb 8 Mb/s - Buff 384 kb Bikes 16 Mb/s - Buff 768 kb Table Tennis 1 Figure54:VBVFullnessinIntra-frameOnlyCBREncoding. 9 8 7 6 5 4 3 2 73 1 3 6 9 12 15 3 6 9 12 15 VBV fullness (%)

tothesmallvalueofthequantizationparameter(seefigure52).theglobaldistortionlevel Inthe\hockey"sceneat16Mb/sthedimensionofadjacentpicturesvariesverymuchdue Figure55:AmountofBitsEncodingEachPicture. T 2T 3T 4T 5T 6T Time I Frame Figure55andconrmedbytheexperimentalresultsshowninFigure2.Thus,ifpredictive Gisanintegerandisvariedby1;duetothelowvalueofG,therelativevariationislarge Predictivecodingyieldsbettercompressionthanintra-framecoding,asqualitativelyshownin andisreectedinalargevariationofpicturedimension. 5.2Intra-frameandPredictiveCoding P Frame codingisusedtoencodeascenewithagivenvisualquality,thewholenumberofbitsproduced issmallerthanifonlyintra-framecodingisused;thelargerthegop(i.e.,thenumberofpframesperi-frame),thesmallertheamountofbitsproduced13.moreover,theslowerthe predictivecodingmustbeusedtoincreasecompression. scene,thesmallerthedimensionofp-frames;sincevideoconferencingscenesarelikelyslow, thecodingshapingdelayislargerthanwhenusingonlyintra-framecoding.then,weshow slowerthesequence,thehigherthevisualqualityoftheencodedsequence.nevertheless, showthatwhenacbrmpegencoderusespredictivecoding,giventhetargetrate,the ofcodingshapingdelay,werstshowwherepicturesexperienceit.insection5.2.2we unsuitableforvideoconferencingapplications.inordertoprovideabetterunderstanding isstudied.weshowthatcodingshapingdelaymakescbrmpegwithpredictivecoding InthefollowingofthisSection,theimpactofpredictivecodingoncodingshapingdelay howpredictivecodingincreasesthecodingshapingdelaywhenencodingatconstantquality. notdecreasetheamountofbitgeneratedbecausethepredictionerroraccumulatesandrequiresmanybitsto Lastly,theimpactofpredictivecodingonthestartupshapingdelayisconsidered. 13Actually,asshownin[19],thereisamaximumGOPdimensionbeyondwhichenlargingtheGOPdoes beencoded. 74

Processed Basic Encoder Output CBR Encoder Output Bits Figure56:NaturalandTargetGenerationofBitsandEncoderandDecoderBuerFullness. (I) T 1 (P) 2T 2 (P) 3T 3 (P) 4T 4 (I) 5T Time (I) 2 (P) 3 (P) 1 (P) buer,asexplainedindetailinthissection. T 2T S c 3T S c +T 4T S c +2T 5T Time byacbrencoder/decodersystemusingbothintra-frameandpredictivecoding.theupper 5.2.1BueringIssuesRelatedtotheCodingShapingDelay diagramqualitativelyshowsthati-framesarelargerthanp-frames.thenaturalrateaveraged amountofbits.picturesexperiencepartofitintheencoderbuerandpartinthedecoder InSection5.1.1ithasbeenshownthataCBRMPEGencoder/decodersystemintroducesa codingshapingdelayinordertoallowpicturestobeencodedatconstantratewithvariable Figure56showshowthebitsofanencodedvideostreamareproducedandbuered Buffer onavideoframeperiodwhilethebasicencoderisintra-framecoding,islargerthanthetarget Buffer rate;thenaturalratewhilepredictivecodingissmallerthanthetargetrate. theirdimensionisthemaximumallowedbythesystem.forexample,picture,whichis assoonastheyarrivebecausetheyexperiencetheprocessingresynchronizationdelayunless correspondingtoaframe.encodedpicturesarenotretrievedfromthebueranddecoded bitsproducedbythebasicencoderforthepreviouspicturesandstillnotsentoutbythecbr encoder.thedecoderbuerlevelisincrementedlinearlyanddecrementedbyretrievalunits (dashedline)buer.p-framesndabacklogintheencoderbuerduetothehighnumberof Thelowerdiagramplotsthefullnessleveloftheencoder(continuousline)andthedecoder 75

butitisnotdecodeduntiltheinstantsc,i.e.,untilthecodingshapingdelayiselapsedfrom itscapture. 5.2.2CodingShapingDelayatConstantTargetRate Weconsidertheencoderoperatingataxedtargetrateandstudytheeectofthecoding intra-framecoded,fullyentersthedecoderbuer(exitstheencoder)atthetimetagged(i), ordertohaveasimple(eventhoughnotaccurate)measureofthevisualqualitywhichallows dierentcongurationtobecompared,wemakethefollowingassumption. aresmallandmorebitsoutofthegopbudgetareusedinencodingi-frames.ifascene MPEGencodersusuallyhavetheirxedtargetrateandchangetheratiobetweenI-frameand isfast,p-framesrequiremorebitsandthedimensionofi-framesdecreasesaccordingly.in P-framedimensionaccordingtothemotioninthescenebeingencoded.Ifitisslow,P-frames shapingdelayonthevisualqualityoftheencodedstream.thisisarealisticscenarioascbr Assumption4Ifthesamesceneisencodedmorethanoncewithdierentencodingparameters,thevisualqualityoftheencodedstreamsobtainedwiththedierentexperimentscanbe comparedbycomparingthedimensionofi-frames:largeri-framesmeanbettervisualquality. AccordingtothisAssumptionandtowhatstatedabove,giventhetargetrate,thefastmoving encoderproduceslargei-frames.bydoingthisitintroducesalargecodingshapingdelay whichislowerboundbythedimensionofthelargestpictureaccordingtodisequation(17). isascene,thelowerthequality. dimensionandotherparametersofacbrencoderthatexploitspredictivecoding.weassume Inavideoconferencingscenario,scenesareusuallyslowandthus,givenatargetrate,the videoconferencingapplicationsshouldbetternotexploitpredictivecodinginordertoreduce thattheencoderusesaconstantnumberofbitstoencodeeachgop:itmustproducethese bitsinagopperiod,i.e.,eachgopisencodedwithntbbits.actualcbrdonot shapingdelayofanyparticularencoder.nevertheless,itgivestheavorthatcbrencodersfor usuallycomplytothisassumption;thus,theexpressionwedevisedoesnotprovidethecoding WenowdeviseanexpressionthatgivesthecodingshapingdelayasfunctionofP-frame theend-to-enddelay. wherefiisthedimensionofthei-frameinthegop,fp ByextractingFIfromEquation(21),weobtain,foreachGOP, Accordingtotheforegoingassumption,thefollowingequationholdsforeachGOP FI+N?1 Xi=1FP i=ntb 76 iisthedimensionoftheithp-frame. (21)

ThesmallerthenumberofbitsusedtoencodeP-frames,thelargerthedimensionofthe I-frameintheGOP. FI=NTB?N?1 Xi=1FP codingisexploited,thelargestpictureisthei-frameinthegopwiththesmallestp-frames. maxseqfisthedimensionofthelargestencodedpictureinthesequence.whenpredictive Equation(18)inSection5.1.1givesthecodingshapingdelaySc=maxseqF=B,where i (22) dimension,i.e.,maxseqf=iasgivenbyequation(22);substitutinginequation(18): Forthesakeofsimplicity(alsointhenotation),weassumethateachI-framehasthesame theencoding/decodingsystemmustintroduce.iftheencodedsceneiscompletelystatic,the PictureDimensionThesmallertheP-frames,thelongerthecodingshapingdelaythat P-framesareencodedwithaverysmallamountofbits;inprinciple.Thus,theI-frame Sc=NTB?PN?1 B i=1fp i=nt?pn?1 i=1fp B i toacodingshapingdelayequaltothegopperiodnt. growsusingallthebitsintendedfortheencodingofthegop,i.e.,ntb;thiscorresponds (23) bitsusedtoencodepicturesisthevbvdimensionvs;adimensionequaltontballows slowlymovingandwouldallowalargeratio.thekeyelementindeterminingthenumberof theratiobetweenthedimensionofi-framesandp-frames,eventhoughthescenetoencodeis theencodertotakefulladvantageofpredictivecodingwhenthesceneisparticularlyslowand tousealargeamountofbitstoencodei-frames.alargerdimensionallowstheencoderto deliverevenmoreuniformqualitybysmoothingsuddenincreaseinthecomplexityofpictures Thus,aCBRencoderwhichaimsatyieldingashortcodingshapingdelaymustbound andinthemotionovermorethanagopperiod.nevertheless,thisrequireslargercoding shapingdelayandbuersinbothencoderanddecoder. P-frames.Thisisparticularlyusefulwhenaparticularlycomplexpicture(i.e.,withlowspatial usedintheencodingofeachgopandapreferredratiobetweendimensionofi-framesand redundancy)isbeingintra-framecoded;iftheencoderusedonlythevbvtodeterminethe bitsusedtoencodepictures.forexample,thesoftwareencoderdvdencbeforeencodinga targetforthepicturedimension,inordertodelivergoodquality,itwouldusethemaximum picturesetsatargetdimension;thisischosenaccordingtoapredenedamountofbitstobe BesidetheVBV,CBRencodersbetterhaveothermeansforcontrollingthenumberof numberofbitsallowedbythesystemtoencodeapicture,i.e.,itwouldllupthevbv. AccordingtoClaim2,thedimensionoffollowingpicture(s)wouldbeboundbyBT;this boundcanbetoosmall,particularlyifthescenebecomessuddenlyfast.infact,p-framesare encodedwithanumberofbitsthatisnotsucienttoshowthesamevisualqualityofthe I-framecorrespondingI-frameandthusthequalityoftheGOPislowered. 77

Nevertheless,Equation(22)showsthatincreasingthevideoframeperiodalsoimprovesthe theencodedscene,itshouldbeavoidedtokeepthedelaysmaller14. thetargetrate;eventhoughaccordingtoassumption4thisincreasesthevisualqualityof GOPSizeIncreasingtheGOPsize(i.e.,N)increasesthecodingshapingdelaybecausethe qualityoftheencodedscenebecausei-framesbecomelarger. byequation(23);thelongerthevideoframeperiodt,thelargerthecodingshapingdelay. percentageofsmallpictures(thep-frames)increasesandsoi-framesaremadelargertokeep Conclusion1GiventhetargetrateofaCBRMPEGencoder,predictivecodingincreasesthe visualquality.nevertheless,everyparametersettingaimedatqualityimprovementincreases Usingalowvideoframerate(i.e.,largeT)increasesthecodingshapingdelay,asshown qualitythecbrencodermustdeliver.weassumetoencodethesamescenemorethanonce Wenowstudytheimpactofpredictivecodingonthecodingshapingdelay,giventhevisual thecodingshapingdelay. 5.2.3CodingShapingDelayatConstantVisualQuality thani-framesandasmuchgopsizenislarger. I-frames,accordingtoAssumption4;weneedawaytocomparetheeectivenessofpredictive coding.theexploitationofpredictivecodingisasmoreeectiveasmuchp-framesaresmaller exploitedandthevisualqualitydeliveredisthesame. theparametersoftheencoderarechangedsothatpredictivecodingismoreorlessheavily withdierentamountsofcompressionduetopredictivecoding.i.e.,foreachexperiment usingadierentnumberofp-framespergop,thei-framesproducedineachexperimenthave thesamedimensionfi.assumption4isaroughapproximation,butitisclosetorealitywhen AccordingtoAssumption4,ifthesamesceneisencodedwithsamevisualquality,but Thevisualqualityoftheencodedstreamsiscomparedbycomparingthedimensionof withconstantamountofbitspandexploitasmeasureofyieldedcompressiontheratio thecontextoftheanalysiscarriedoninthissection,weassumethatp-framesareencoded dealingwithsimpleslowmovingscenes(suchasatypicalvideoconference);moreover,theaim ofthissectionisonlytogivetheavoroftheimpactofpredictivecodingonthecodingshaping delay,nottoprovideageneralanalyticalrelationshipbetweenthetwo.insection4.1.2we statedthatusuallythedimensionofi-framesisfrom2to4timesthedimensionofp-frames; theslowertheencodedscene,thelargertheratiobetweeni-frameandp-framedimension.in 14IncreasingtheGOPsizealsorenderstheencodedstreammoresensitivetoerrorsandlosses. Therateoftheencodedstreamcanberoughlyexpressedas =FI 78FP

wherenisthenumberofpicturespergopandfpisp-framedimension,whichweassume tobeconstantforallthep-frames.equation(24)canberewrittenas BN=FI+(N?1)FP BN=N+?1 NTFI (24) Bymergingthisequationwith(25),thelattercanbewrittenas Tobetterhighlighttheeectofpredictivecodingontherateoftheencoder,wealsoprovide theencoderratewhenonlyintra-framecodingisused: BI=FI T (26) (25) DuetotheapproximationsmadetodevisethisEquation,itdoesnotexactlyreectthereal behavior,butitgivestheavorthatthemoreeectivepredictivecoding(i.e.,theratio),the smallerthetargetrateneededtodeliverthesamevisualquality.also,thelargerthenumber ofp-framespergop(n),thesmallerthetargetrate. BN=BIN+?1 Equation(18)inSection5.1.1givesthecodingshapingdelayasSc=maxseqF=B,where Wenowanalyzetheeectofpredictivecodingatconstantqualityoncodingshapingdelay. N (27) BeingFIconstantwhenthequalityisconstant,thetargetrateoftheencoderBistheonly maxseqfisthemaximumdimensionofframesandbisthetargetrateofthecbrencoder. thevalueofbwithbngivenbyequation(25) factorwhichdeterminesthecodingshapingdelayindierentcongurations.bysubstituting AccordingtoAssumption4,maxseqF=FIandthecodingshapingdelayisexpressedas Sc=NT Sc=FI N+?1 B Themorepredictivecodingiseective(i.e.,),thelargerthecodingshapingdelay.Typical.Inavideoconferencingapplication,itislikelytohave=4;withN=15at15fps, valuesofarebetween2and4;theslowerthemotioninthescene,thelargerthevalueof largerthanthe1msboundessentialforinteraction.similarconsiderationsapplyalsoto Sc=222ms,i.e.,theCBRencoderaloneintroducesinthevideoconferencingsystemadelay theincreaseofn. 79

Processed Basic Encoder Output CBR Encoder Output Bits Conclusion2UsingaCBRMPEGencoderwithpredictivecodingtobuildavideoconferencingsystem,reducesthebandwidthrequirement,butitisnotacceptablefromthepointofview (I) T 1 (P) 2T 2 (P) 3T 3 (P) 4T 4 (I) 5T Time Bits (I) 2 (P) 3 (P) 5.2.4StartupShapingDelay oftheend-to-enddelay. Figure57:RealisticBitGeneration. 1 (P) AsshowninFigure57,atypicalMPEGencoderencodesI-frameswithalargeamountofbits T 2T S c 3T S c +T 4T S c +2T 5T Time inasmalltime,withrespecttop-frames.asaconsequence,itisunlikelythatthenatural ratebelowerthanthetargetratewhentheencodingofagopbegins.after,thebuerhas Encoder Buffer Decoder Buffer quantizationstepsizeinordertoincreasethenaturalrate)arevisibleonasmallpercentage thehighestunderowprobabilityisafterthewhenthebuergetsemptyattheendofa GOPwithsmallP-frames.Duringtherestofthetimeabacklogislikelytobepresentin delayisunderdimensioned,theeects(nonuniformqualityduetoasuddendecreaseofthe theencoderbuer,ascanbeseeninfigure56.thus,eventhoughthestartupshaping somebacklogduetothelargeamountofbitsencodingthei-frame. ofthepictures. TheencoderbuercanhappentounderowonlywhenP-framesareparticularlysmall; 8

oftheratecontrolfunction.thegopdimensionissetton=15,i.e.,eachi-frameisfollowed valuethatguaranteesthevbvnottobeoverowed,notwithstandingthelowaggressiveness threedierentrates.theratiobetweenthevbvdimensionandtherateiskeptconstanttoa Figures58-61showtheexperimentaldataobtainedbyencodingthefoursamplesequencesat by14p-frames. 5.3ExperimentalData dimensionofi-framesandp-frames.thisisduetothefactthatthecameraispanningover thehockeycourttofollowthegame.picturesaresimple(mostofthescreenisthewhiteice andthedimensionofbothi-framesandp-framesisvariable.especiallyatlowbitratesthe dimensionofi-framesisdoubleormorethandoublethedimensionofp-frames:inorderfor thistobepossiblethecodingshapingdelaymustbelarge. Inthe\hockey"sequenceencodedat16Mb/sthereisalmostnodierencebetweenthe Figure58showsthedimensionofeachpicture.I-framesaretypicallylargerthanP-frames isdecreased;beingitsmall,decreasingitbyoneunithasasignicanteectonthepicture dimension:thesecondp-frameismuchsmallerthantherstone.thiscausesthevbv smallerthanthei-frame.inordertoreducep-framedimensionthequantizationparameter leveltodecrease(figure61)andthequantizationparametertobereducedagaintoavoid thewholeimagesandp-framesdimensionislarge:therstp-frameinthegopisnotmuch asmallquantizationparameter(figure59).sincethecameraispanning,motionconcerns ofthecourt)sothati-framescanbeencodedwithfewbits;thep-framesarequantizedwith fullandemptiesmoreslowly(bottomrightplotinfigure65). underow.thisbehavioriskeptforthewholegopthusleadingtotheoscillationinpframesdimensionandpicturequality(seesnrinfigure6).figure59showstheaveragenceisusuallycharacterizedbylowmotionandpictureshavinglowcomplexity.thus,when whichi-framedimensionincreases(bottomrightplotinfigure62)andthevbvgetsmore arequitesimilartotheoneobtainedwithn=15exceptforthe\tabletennis"sequencein quantizationparametervalueusedoneachframe. compressingthemwithacbrmpegencoderafairlysmallamountofbitscanbesucient Thescenesusedintheseexperimentsarenotvideoconferencingsequences;avideoconfer- Figures62-65showtheexperimentaldataobtainedusingaGOPof3pictures.Theresults toencodei-frameswithgoodquality;whenencodingap-frametheerrorbetweenthemb thatisbeingencodedandthereferencembissmall,andthustheamountofbitsrequiredis lowinordertoemphasizethisbehaviorofacbrmpegencoder,weexploitedtoencodea applicationif,forexample,thecameraispointedatawhiteboardwhilesomebodyisexplaining itscontent.weobtainedtwostaticscenesbyrepeating12timesthesameimageofthe staticscene.thisisanextremecaseofslowmovingscenebutislikelyforavideoconferencing \cheerleaders"and\hockey"scenes,respectively. Figure66showsthedimensionofframesforvarioustargetratesandVBVdimensions. 81

Cheerleaders Hockey Picture dimension (Kb) 12 1 8 6 4 2 Picture dimension (Kb) 12 1 8 6 4 2 3 6 9 12 15 4 Mb/s - Buff 2.4 Mb 8 Mb/s - Buff 4.8 Mb 16 Mb/s - Buff 9.6 Mb Bikes 3 6 9 12 15 Table Tennis Picture dimension (Kb) 12 1 8 6 4 2 Figure58:DimensionofPicturesinCBRMPEGEncodingwithN=15. 12 1 8 6 4 82 2 3 6 9 12 15 3 6 9 12 15 number number Picture dimension (Kb)

Average Quantization Parameter Average Quantization Parameter 8 7 6 5 4 3 2 1 Cheerleaders Average Quantization Parameter 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 2.4 Mb 8 Mb/s - Buff 4.8 Mb Bikes 16 Mb/s - Buff 9.6 Mb Table Tennis 8 Figure59:AverageQuantizationParameterinCBRMPEGEncodingwithN=15. 8 7 7 6 6 5 5 4 4 3 3 2 2 1 83 1 3 6 9 12 15 3 6 9 12 15 Average Quantization Parameter 8 7 6 5 4 3 2 1 Hockey

5 Cheerleaders 5 Hockey 45 45 SNR (db) 4 35 SNR (db) 4 35 3 3 SNR (db) 25 5 45 4 35 3 25 25 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 2.4 Mb 8 Mb/s - Buff 4.8 Mb Bikes 16 Mb/s - Buff 9.6 Mb Table Tennis 3 5 Figure6:SNRinCBRMPEGEncodingwithN=15. 45 4 35 3 84 25 9 12 15 3 9 12 15 SNR (db)

35 Cheerleaders 35 Hockey 3 3 VBV Fullness (%) 25 2 15 1 VBV Fullness (%) 25 2 15 1 5 5 VBV Fullness (%) 35 3 25 2 15 1 5 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 2.4 Mb 8 Mb/s - Buff 4.8 Mb Bikes 16 Mb/s - Buff 9.6 Mb Table Tennis 35 Figure61:VBVFullnessinCBRMPEGEncodingwithN=15. 3 25 2 15 1 85 5 3 6 9 12 15 3 6 9 12 15 VBV Fullness (%)

Picture dimension (Kb) Picture dimension (Kb) 16 14 12 1 8 6 4 2 16 14 12 1 8 6 4 2 Cheerleaders Picture dimension (Kb) 16 14 12 1 8 6 4 2 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 2.4 Mb 8 Mb/s - Buff 4.8 Mb Bikes 16 Mb/s - Buff 9.6 Mb Table Tennis 16 Figure62:DimensionofPicturesinCBRMPEGEncodingwithN=3. 14 12 1 8 6 4 86 2 6 9 12 15 6 9 12 15 number number Picture dimension (Kb) Hockey

Average Quantization Parameter Average Quantization Parameter 8 7 6 5 4 3 2 1 Cheerleaders Average Quantization Parameter 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 2.4 Mb 8 Mb/s - Buff 4.8 Mb Bikes 16 Mb/s - Buff 9.6 Mb Table Tennis 8 Figure63:AverageQuantizationParameterinCBRMPEGEncodingwithN=3. 8 7 7 6 6 5 5 4 4 3 3 2 2 1 87 1 3 6 9 12 15 3 6 9 12 15 Average Quantization Parameter 8 7 6 5 4 3 2 1 Hockey

5 Cheerleaders 5 Hockey 45 45 SNR (db) 4 35 SNR (db) 4 35 3 3 SNR (db) 25 5 45 4 35 3 25 25 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 2.4 Mb 8 Mb/s - Buff 4.8 Mb Bikes 16 Mb/s - Buff 9.6 Mb Table Tennis 3 5 Figure64:SNRinCBRMPEGEncodingwithN=3. 45 4 35 88 25 6 9 12 15 6 9 12 15 SNR (db)

VBV Fullness (%) VBV Fullness (%) 7 65 6 55 5 45 4 35 3 25 2 15 1 5 7 65 6 55 5 45 4 35 3 25 2 15 1 5 Cheerleaders VBV Fullness (%) 7 65 6 55 5 45 4 35 3 25 2 15 1 5 Hockey 3 6 9 12 15 3 6 9 12 15 4 Mb/s - Buff 2.4 Mb 8 Mb/s - Buff 4.8 Mb Bikes 16 Mb/s - Buff 9.6 Mb Table Tennis 7 Figure65:VBVFullnessinCBRMPEGEncodingwithN=3. 6 55 5 45 4 35 25 2 15 89 1 5 6 9 12 15 6 9 12 15 VBV Fullness (%)

25 Cheerleaders - VBV 1.6 Mb 25 Hockey - VBV 1.6 Mb Picture dimension (Kb) 2 15 1 5 Picture dimension (Kb) 2 15 1 5 25 3 6 9 12 4 Mb/s 8 Mb/s Cheerleaders - 16 Mb/s 16 Mb/s 25 3 6 9 12 Hockey - 16 Mb/s Picture dimension (Kb) 2 15 1 5 Picture dimension (Kb) 2 15 1 5 Picture dimension (Kb) 3 6 9 12 3 6 9 12 VBV 1.6 Mb VBV 3.2 Mb VBV 6.4 Mb 4 Mb/s - VBV 1.6 Mb 8 Mb/s - VBV 3.2 Mb Figure66:DimensionofPicturesinCBRMPEGEncodingofaStaticScene Cheerleaders 16 Mb/s - VBV 6.4 Mb Hockey 25 25 9 2 2 15 15 1 1 5 5 3 6 9 12 3 6 9 12 number number Picture dimension (Kb)

spondingnonstaticsceneisencodedatthesametargetrate.p-framesaresmallbecausethey mustonlyencodethedierencebetweentheirreferencepictureandtherawpicturethatis duetothequantization.thus,thesnrdeliveredbyp-framesishigh,asshowninfigure68. InanyplottheratiobetweenI-frameandP-framedimensionislargerthanwhenthecorre- parameterhastheminimumallowedvalueandcannotbefurtherreduced(seefigure67). dimensions(secondrow,rstcolumninfigure66)showsthatthelargeristhevbvdimension, allowalargeramountofbitstobeproduced,i-framesdonotgrowbecausethequantization thelargestarethei-frames.thisdoesnotholdforthe\hockey"scene(secondrow,second column)because,beingthepictureslessdetailed,eventhoughtheratecontrolfunctionwould Theplotobtainedbyencodingthe\cheerleaders"sceneat16Mb/swithvariousVBV thefullnesslevelisoftheorderofthe2%.nevertheless,whendealingwithreal-time inadvance.ifthevideoconferencingsystemmustbeabletodeliveragoodquality,especially encoding(suchasinvideoconferencingapplications),thescenetobeencodedisnotknown whenthecameraisstaticallysetonsomeimage,thevbvmustbelargeenoughtoallow I-framestogrowandcapturethedetailsoftheimage15(e.g.,secondrow,rstcolumnin Figure69).Whenpicturesarenotdetailed(e.g.,secondrow,secondcolumninFigure69)or Figure69showsthatinmanycasestheVBVis(almost)completelylled,whileinothers themotionishigher(figure61)onlyafractionofthevbvislled.inanycase,thecoding shapingdelayislargebecauseitmustbesetaccordingtothevbvdimensioninorderto 5.4DedicatedLinkbetweenSenderandReceiver guaranteecontinuousplaying. Forthesakeofcompletenessofthisstudy,thesystemcongurationinwhichsenderand receiverareconnectedthroughadedicatedlink(figure7)istakenintoaccount.havingthe advantages.onlyafractionofthelinkcapacity,correspondingtothetargetratebofthe encoderisactuallyexploitedandthecodingshapingdelayprovidesasignicantcontribution totheend-to-enddelay. wholelinkcapacityatdisposalforsendingthecbrmpegstreamdoesnotprovideparticular systemexploitingcbrmpegencoding. theend-to-enddelayofthesystemis lay.thisrepresentalowerboundintheend-to-enddelayobtainablefromavideoconferencing wheredisthedecodingdelay,scisthecodingshapingdelay,andssisthestartupshapingde- GiventhepropagationdelayP,whichdependsonthedistancebetweensenderandreceiver, 15Notethatasthesceneisnotmoving,thehumaneyeismoresensitivetoerrors. Ded CBR=Sc+Ss+P+D+Pd (28) 91

Cheerleaders - VBV 1.6 Mb Hockey - VBV 1.6 Mb Average Quantization Parameter 4 35 3 25 2 15 1 5 Average Quantization Parameter 4 35 3 25 2 15 1 5 3 6 9 12 4 Mb/s 8 Mb/s Cheerleaders - 16 Mb/s 16 Mb/s 3 6 9 12 Hockey - 16 Mb/s Average Quantization Parameter 4 35 3 25 2 15 1 5 Average Quantization Parameter 3 6 9 12 3 6 9 12 VBV 1.6 Mb VBV 3.2 Mb VBV 6.4 Mb 4 Mb/s - VBV 1.6 Mb 8 Mb/s - VBV 3.2 Mb Figure67:AverageQuantizationParameterinCBRMPEGEncodingofaStaticScene Cheerleaders 16 Mb/s - VBV 6.4 Mb Hockey 4 92 4 35 35 3 3 25 25 2 2 15 15 1 1 5 5 3 6 9 12 3 6 9 12 Average Quantization Parameter Average Quantization Parameter 4 35 3 25 2 15 1 5

Cheerleaders - VBV 1.6 Mb Hockey - VBV 1.6 Mb 5 5 SNR (db) 45 4 SNR (db) 45 4 35 35 3 6 9 12 4 Mb/s 8 Mb/s Cheerleaders - 16 Mb/s 16 Mb/s 3 6 9 12 Hockey - 16 Mb/s 5 5 SNR (db) 45 4 SNR (db) 45 4 35 35 SNR (db) 5 45 4 35 3 6 9 12 3 6 9 12 VBV 1.6 Mb VBV 3.2 Mb VBV 6.4 Mb 4 Mb/s - VBV 1.6 Mb 8 Mb/s - VBV 3.2 Mb Figure68:SNRinCBRMPEGEncodingofaStaticScene Cheerleaders 16 Mb/s - VBV 6.4 Mb Hockey 93 5 45 4 35 3 6 9 12 3 6 9 12 SNR (db)

VBV Fullness (%) VBV Fullness (%) VBV Fullness (%) 1 9 8 7 6 5 4 3 2 1 1 9 8 7 6 5 4 3 2 1 1 9 8 7 6 5 4 3 2 1 Cheerleaders - VBV 1.6 Mb VBV Fullness (%) 1 9 8 7 6 5 4 3 2 1 3 6 9 12 4 Mb/s 8 Mb/s 16 Mb/s Cheerleaders - 16 Mb/s 1 9 8 7 6 5 4 3 2 1 VBV Fullness (%) Hockey - VBV 1.6 Mb 3 6 9 12 Hockey - 16 Mb/s 3 6 9 12 3 6 9 12 VBV 1.6 Mb VBV 3.2 Mb VBV 6.4 Mb 4 Mb/s - VBV 1.6 Mb 8 Mb/s - VBV 3.2 Mb Figure69:VBVFullnessinCBRMPEGEncodingofaStaticScene Cheerleaders 16 Mb/s - 6.4 Mb Hockey 1 94 9 8 7 6 5 4 3 2 1 3 6 9 12 3 6 9 12 VBV Fullness (%)

Figure7:ArchitectureofaVideoconferencingSystemExploitingaCBRMPEGEncoder Figure71:ArchitectureofaVideoconferencingSystemExploitingaCBRMPEGEncoder andadedicatedlinkbetweensenderandreceiver. Frame Network Network Encoder Decoder Grabber Interface Interface Adaptor rateoftheencoder.thecontributionofthenetworktotheend-to-enddelayistheswitching MPEGencoderisusedtobuildavideoconferencingsystem(Figure71),themosteectiveway toconnectsenderandreceiveristhroughacircuitthathasabandwidthequaltothetarget CBRMPEGencodingissuitedfortransmissionovercircuitswitchednetworks:whenaCBR andacircuitswitchednetwork. Frame Encoder receiver.theend-to-enddelayofthesystemis 5.5CircuitSwitching delayswandthepropagationdelaypwhichdependsonthedistancebetweensenderand Decoder Interface Network Grabber Interface Adaptor systemexploitingcbrmpegencoding. wheredisthedecodingdelay,scisthecodingshapingdelay,andssisthestartupshaping representsaswellalowerboundintheend-to-enddelayobtainablefromavideoconferencing receiver:theonlyfurthercontributionistheswitchingdelay.thus,itcanbesaidthatit delay.thisdelayalmostequalstheoneobtainedwithadedicatedlinkbetweensenderand CS CBR=Sc+Ss+Sw+P+D+Pd (29) 5.6PacketSwitchingwithTimeDrivenPriority CBRMPEGencodingisbestsuitedfortransmissionovercircuitswitchednetworks.Nevertheless,inthefollowingtwoSectionsweconsidertheend-to-enddelayobtainedwhenCBR encodedvideoissentthroughapacketswitchednetwork.inthissection,timedrivenpriority senderandreceiverisanalyzedrst;then,theend-to-enddelayisconsidered. isconsideredtobeexploitedinthenetwork.thenetworkshapingdelayintroducedinthe 95

Frame Grabber Video Adaptor Encoder Decoder Packetization Packetization Figure72:ArchitectureofaVideoconferencingSystemExploitingaCBRMPEGEncoder andapacketswitchednetworkwithtimedrivenpriority. 5.6.1NetworkShapingDelay Whentimedrivenpriorityisusedinthenetwork,thetimecycleissettoTc,thevideoframe period,andbtcbitsmustbereservedineachtimecycle.thebitsexitingthecbrmpeg encoderarebueredinthebuerofthepacketizationfunctiondepictedinfigure72,untilthey canbeincludedinapacket.eachbitexperiencesadierentdelay,butduetorequirement3 ofcontinuousplaying,anetworkresynchronizationdelaymustbeintroducedonthereceiver sidesothateachbitexperiencesthesamewholedelayspack nwhichiscalledthepacketization componentofthenetworkshapingdelay. Thepacketizationfunctiononthereceiversideintroducesthenetworkresynchronization delay;thepacketizationfunctionknowsthedelayexperiencedinthebuerofthesender packetizationfunctionbythebitsofeachreceivedpacketandcancompensateit.thecompensationisdonebybueringthebitsandprovidingtothedecoderabitowattheconstant rateb.ifthetfsreservedtothevideoconferencecallareevenlydistributedinthetime cycle,nonetworkresynchronizationisrequired. ThedimensionofthebuerofbothpacketizationfunctionsmustbeatleastBSPack n. PPack nisthemaximumtimeabitspendsinthesenderbuerbeforebeingretrievedand insertedinapacket;thebuer,whichislledatconstantrateb,doesnotoverowifit canstorethebspack nbitsgeneratedbythedecoderinthemeantime.spack nisaswellthe maximumtimespentbyabitinthereceiverbuer;thebuercannotcontainmorethan BSPack nbitsotherwisethelastonewouldtakemorethanspack ntoexitattheconstantrate B.TheactualvalueofSPack ndependsonhowthetfshavebeenreservedwithinthetime cycleforthevideoconference;thus,itisxedatresourcereservationtimeandisconstant overthewholevideoconferencecall(oruntilthebandwidthisreallocated).if,forexample, asingletfisreserved,spack n=tcbecausetherstbitexitingthecbrencoderaftera packethasbeenassembledandsenthastowaitawholetimecyclebeforebeinginsertedina packet.intuitively,thesmallerandthemoreuniformlydistributedinsidethetimecycleare thepackets,theshorterthenetworkshapingdelayspack n. Exploitationofsmallanduniformlydistributedpacketsisencouragedalsobyotherarguments.InSection4.3.5ithasbeenshownthat,ifthedecoderstartsdecodingassoonassome ofthebitsencodingapictureenteritsbuer,thedecodingtimeisreducedbyusingsmall packets.in[14]isshownthatreservingasmallamountofbitsinmanytfsisconvenient 96

provideslowerblockingprobability.usingsmallpacketshasafurtheradvantagewhentime drivenpriorityisexploitedinthetransmissionofcbrmpegencodedvideo. alsofromthepointofviewofthenetworkbecausebalancingtheloadalongthetimecycle encodingasinglepicture.thenetworkshapingdelaycanbeminimizedifthetfsarereserved streamiscomposedofpacketshavingxeddimensionof188bytes;eachpacketcontainsbits ordertomakemorerobustitstransmissionoverpacketswitchednetworks.thetransport producedbyacbrmpegencoderbestructuredasatransportstream(seesectionb.5)in sothatassoonasanmpegpacketisproduced,areservedtfisscheduledandthepacketis MPEGstreammustbetakenintoaccount.TheMPEGstandardspeciesthatthebitstream Toidentifytheschedulethatminimizesthenetworkshapingdelay,thestructureofthe sent16.thisrequirestoreserve188bytesintfsequallyspacedby(1888)=bandhavethe encodersynchronizedwiththenetworkinterfacesothatthelastbitofeachmpegpacketis emittedjustbeforeareservedtfisscheduled17.iftheencoderisnotsynchronizedwiththe networkinterface,thereisanapplicationsynchronizationcomponentofthenetworkshaping delaysas components,i.e., Inthegeneralcase,thenetworkshapingdelayisgivenbythecompositionofthetwo n2[;(1888)=b]. 5.6.2ExcessResynchronizationDelay ofthereceiver. anditisexperiencedpartlyinthesenderandpartlyinthenetworkresynchronizationfunction Sn=SAS n+spack Inwhathasbeendiscussedsofar,thenetworkresynchronizationdelayisassumedtobe n introducedbyanadhocfunctionwhichcanbeimplemented,forexample,inthesoftware layersbelowtheapplication.nevertheless,itcanalsobeintroducedinthedecoderbuer; mustbeenlarged.ifthedecoderissynchronizedwiththenetworkinterface,thebuermust bueringisnotrequiredinthereceiversoftwarebelowthedecoder18andthedecoderbuer end-to-enddelaybecauseitisalreadytakenintoaccountbythecodingshapingdelay:thebitswouldbe untilallthe188byteshavebeensentoutoftheencoder.however,thistimedoesnotcontributetothe onlyvariablecomponentofthenetworkdelay. isexploitedinthenetwork,thepacketizationcomponentofthenetworkshapingdelayisthe beenlargedby,atleast,bspack 17Actually,eveninthissituation,thebitsofeachpackethastobebueredinthepacketizationfunction 16BreakinganMPEGpacketandsendingitduringtwoTFsdecreasesrobustnesstolossesanderrors. n.thisstemsfromthefactthatwhentimedrivenpriority anywaykeptinthedecoderbuertointroducetheprocessingresynchronizationdelay.theonlyimpacton theend-to-enddelayisapossibleincreaseofthedecodingtimeifthedecoderhasanaccessunitsmallerthan thempegpacket. notrelevanttothiswork. 18Actually,abuerisneededonlytobuereachincomingpacketwhiletheheaderisprocessed,butthisis 97

andthedecoderbuermustbeenlargedby,atleast,bspack partlyinthesenderpacketizationfunction,partlyinthedecoderbuerisactuallyspack accordingly19.iftheassumptionistrue,thenetworkshapingdelayexperiencedbyeachbit thebitshaveexperiencedtheminimumdelayandthedecodingofpicturesmustbedelayed receivedthedecoderdoesnotknowhowmuchtimethebitshavespentinthebuerof thesenderpacketizationfunction.thus,thedecoderusestheconservativeassumptionthat Ifthedecoderisnotsynchronizedwiththenetworkinterface,whentherstpacketis true,aexcessresynchronizationdelayerisintroduced,i.e.,bitsarebueredlongerthan receivedpackethasalreadyexperiencedadelayspack totheexcessbueringtimeer.inthefollowing,themaximumvalueoferandtheminimum necessary;thedecoderbuermustbedimensionedaccordinglysothatitdoesn'toverowdue dimensionofthebuerarecalculated. Thelargestexcessresynchronizationdelayisintroducedwhentherstbitintherst n.iftheassumptionisnot n function.thedecodershouldnotintroduceanynetworkresynchronizationdelay;insteadthe buer,itisnecessarytocomputetheminimumdelayexperiencedinthesenderpacketization decodingisdelayedbecauseitisassumedthatthebitsexperiencedminimumdelayinthe packetizationfunction.inordertocalculatethetimespentbytherstpacketinthedecoder Bitsexperienceminimumdelayduetopacketization,ifapacketofdimensionPfissentas ninthebuerofthesenderpacketization whentherstbitwasproducedbythecbrencoder,thenetworkresynchronizationdelayexperiencedinthedecoderbuer(whichisalsotheupperboundontheexcessresynchronization delayer)is soonasthecbrencoderemitsthebitnumberpf.inthiscase,therstbitinthepackethad beeninthebuerofthesenderpacketizationfunctionforthetimethecbrencodertakes n?pf B nfrom toproducepfbits,i.e.,pf=b.sincedecodingoftherstpacketmuststartafterspack where andexcessresynchronizationdelayisthusgivenby Theoveralltimespentbybitsinthedecoderbuerduetonetworkresynchronizationdelay n+er (3) networks. resynchronizationdelaytocompensatethevariationofthequeueingdelayinasynchronouspacketswitched 19AsexplainedinSection3.4.1,thereplaybuershowsthesamebehaviorwhenintroducingthenetwork Er2;SPack n?pf B 98

itmustbeenlargedby thenetworkresynchronizationdelayandberobustagainsttheexcessresynchronizationdelay, Thedecoderbuermustbeoversizedaccordingtotheworstcase,i.e.,inordertointroduce call.thus,spack decoderdoesnotknowtheactualvalueofspack TheactualvalueofSPack n2[;tc]dependsonthetfsreservedforthevideoconference 2SPack casespack withamechanismtocommunicateittothedecoder.ifsuchamechanismisnotavailable,the n=tc;i.e.,decodingisstartedafter nisknownatreservationtimebythenetworkinterface;thiscanbeprovided n?pf nanddelaysdecodingaccordingtotheworst BB excessdelayisintroducedwhenthebitsintherstpackethasexperiencedminimumdelay fromthereceptionoftherstpacket.iftheactualpacketizationcomponentofthenetwork excessresynchronizationdelay.inparticular(31)isthemaximumervalue;theminimum shapingdelayisnull(becausetfshadbeenallocatedoptimally)andtherstpackethas experiencedspack ninthesender,itisunusefultodelaydecodingby(31),i.e.,thisvalueisthe Tc?Pf B anddecodingshouldbedelayedby(3): Thedecoderbuermustthusbeenlargedby Er2Tc?SPack SPack n+tc?pf n;tc?pf BBB tothenetworkinterface(i.e.,itdoesnothaveinformationaboutthescheduleofthereserved TFsandtheTFinwhicheachpacketisreceived),theexcessresynchronizationdelayislarger. eliminatestheneedforanadhocbuer.nevertheless,ifthedecoderisnottightlyconnected 5.6.3End-to-endDelay Thisshowsthatnetworkresynchronizationcanbeperformedinsidethedecoder;this where Thegeneralexpressionfortheend-to-enddelayis TDP?SH CBR=Sc+Ss+LTf+Sn+Er+D+Pd 99 (32)

Scisthecodingshapingdelayintroducedbytheencoder/decodersystemtoproduce theencodedstreamattheconstanttargetrate; Ssisthestartupshapingdelayintroducedbytheencodertoavoidtheencoderbuer tooverrunwhenthenaturalrateissmallerthanthetargetrate; LTfisthetimetakenbyapackettotravelthroughthenetworkfromsourcefrom destination,; Sn=SPack n+sas n: {SPack n2[;tc]dependsonthenumber,dimension,andpositionofthetfsreserved tothevideoconferencecallandisconstantoverthedurationofthecall, {SAS n2[;tc]isduetothelackofsynchronizationbetweenencoderandnetwork interface;becauseoftherelativedriftofthetwotimingreferences,thisdelaycan varyinsidethegivenintervalduringthevideoconferencecall; Er2[;Tc?Pf=B]isnotnullwhenthenetworkresynchronizationdelayisintroduced bythedecoderanditisnotsynchronizedwiththenetworkinterface. Ifresourcereservationisoptimal(SPack n=pf=b),theencoderissynchronizedwiththe networkinterface(sas n=),andnetworkresynchronizationisperformedwithinformation fromthenetworkinterface(er=),(32)isverysimilarto(29).thetermssw+pappearing inthelatteraresubstitutedbythetermltfintheformer;allofthesetermsdependon thepropagationdelayandthenumberofnodesonthepathbetweensenderandreceiver. Whenthesenderistopologicallyclosetothereceiver,thecodingshapingdelayScisthemain contributioninbothcs CBRandTDP CBR.Forexample,ifScisaround3Tat3framesper second,andthepropagationdelaypandswitchingdelayswarenegligible,cs CBR'1ms. Ifapacketswitchednetworkwithtimedrivenpriorityisexploitedinthesamescenario,itis likelythatpacketsareforwardedbyonerouter,i.e.,l=3.moreover,ifsmallpacketsare used(pf=188bytes)andtherateofthecbrencoderis1.5mb/s, TDP CBR=CS CBR+SPack n+3tf'1+1+:375'cs CBR Thus,theshortdistancetransmissionofCBRMPEGvideooverapacketswitchednetwork withtimedrivenpriorityyieldsalmostthesameend-to-enddelayasthetransmissionovera circuitswitchednetwork. Ifsenderandreceiverarenotclose(i.e.,thepropagationdelayisnotnegligible),the dierencebetweentdp CBRandCS CBRdependsonthespeedandthenumberofintermediate nodesonthepath.thefasterthepacketswitches,thesmallerthedierence. AsuboptimalscheduleoftheTFsreservedforthevideoconferencecallincreasestheendto-enddelay;nevertheless,suboptimalschedulescanbeacceptedinordertoreducetheprobabilitythatcallsbeblocked. 1

Figure73:MPEGEncoderandPacketizationFunction. Encoder Buffer Packetization Depacketization Decoder Base Encoder Network betweensenderandreceiveristakenintoaccount. areconnectedbyadirectlinkisrstconsidered.then,amulti-hoppacketswitchednetwork theissuesrelatedtopacketization,inthissectionacongurationinwhichsenderandreceiver besegmentedinpacketsandthenreconstructedinthereceiver.inordertobetterfocuson 5.7AsynchronousPacketSwitching TransmissionofaCBRMPEGstreamoverapacketswitchednetworkrequiresthestreamto Control 5.7.1Single-HopPacketNetwork Encoder WiththesystemcongurationdepictedinFigure73,senderandreceiverareconnected throughadirectdedicatedlinkonwhichtheysenddatainpackets,asshownbythepacketizationfunctioninfigure73.thiscongurationisnotnecessarilyusedinrealapplications, Assumption5Transmissionofapacketcannotstartuntilallthebitsthataretobeputin butallowsustofocusonthedelayduetopacketization. packetizationfunction.givenapacketsizeps,whenthebuercontainspsbitsitisemptied inthesystem.thebitsproducedbytheencoderarestoredinthebuerofthesender payload.nevertheless,duetothisassumptionthepacketizationfunctionintroducesadelay headerofthepacketachecksequencethatcanbecalculatedonlyfromthewholepacket itspayloadhasbeenproducedbytheencoder. (asshowninfigure74)andapacketissentoverthenetwork. ThisAssumptionconformswithrealitysincemanyexistingprotocolsneedtoinsertinthe thestructureofthempegtransportstreamsothatasinglempegpacketisnotsentinto twopackets.moreover,weassumethatapacketissenteachtimethelastbitencodinga rateoftheencoderandtokenbucketsizea=ps.packetsizemustbechosenaccordingto senderatregularintervalsps=b,asshowninfigure74.thegeneratedtrachasthesame characteristicsofdataexitingaleakybucketwithtokengenerationratebequaltothetarget pictureexitstheencoder,independentlyfromitsdimension: SincebitsareproducedbytheCBRMPEGencoderatconstantrateB,packetsleavethe 11

Figure74:FullnessoftheBuerofthePacketizationFunction. Bits P s Time 2 P P s 3 s B B B Encoder Output Transmission Assumption6Apacketcontainsbitsencodingonlyonepicture. Thus,therealbehaviorofthepacketizationbuercanbedierentfromtheonedepictedin Figure74,butthebasicprinciplesdiscussedinthisSectionstillhold. Figure75:GenerationandTransmissionofBits;FullnessLeveloftheDecoderBuer. sender.eachbitofapacketexperiencesadierentdelayfromwhenitisgeneratedbythe P s C NoBueringintheDepacketizationFunctionWerstassumethatpacketsgetdi- CBRencodertowhentheyitisputonthelink. AsshownbyFigure75,packetsaresentatthefullspeedCofthelinkconnectedtothe T 2T 3T Time rectlytothedecoderbueruponarrivalatthereceiver,asshownbyfigure76;thedepacke- 12

Decoder Buffer Fullness Bits (I) tizationfunctionhastheonlytaskofstrippingothepacketheader2. P s C 1 (P) togetherwhendecodingstarts.accordingtoassumption6,thelastbitofapictureisthe interestedinthedelayexperiencedbythewholepicture.thesystemtargetdecoderspecied bythempegstandard,thebitsencodingapictureareremovedfromthedecoderbuerall Figure76:FullnessLeveloftheDecoderBuer. P 2 (P) P lastbitinapacket;intheworstcaseatimeps=celapsesfromwhenthislastbitexitsthe s B CBRMPEGencodertowhenitentersthedecoderbuer.Thus,decodingisdelayedbythe packetizationdelaydpthatisthetransmissiondelayofapacket,i.e.: Eachbitofapacketexperiencesadierentdelayduetopacketization;neverthelessweare s T S c +D p 2T S c +D p +T 3T Time buermustnotbeoversized.consideringalsothedelayintroducedbytheencoder/decoder system(sc,ss,andd)andthepropagationdelay,theend-to-enddelayisgivenby Maximumdimensionpacketsexperiencethepacketizationdelayentirelyinthesenderpacketizationfunction;packetsofdimensionP<Psspend(Ps?P)=Cinthedecoderbuer.Since Dp=Ps nofurtherdataarriveduringthistimeandafteritthewholepictureisretrieved,thedecoder ordertopossiblychecktheerrorcontrolcode.thisisnotrelevanttowhatisexplainedinthefollowing:it meansthatfigure76shouldbemodiedbyhavingeachpacketenterthedecoderbuerallatonce. 2Thisassumptionisnotrealisticasthepacketizationfunctionusuallyneedstobuerthewholepacketin Thepacketsizeshallbechosenassmallaspossible(possiblyequaltothedimensionof Async CBR=Sc+Ss+Dp+P+D+Pd (34) 13 C (33)

Encoder Output Transmission Figure77:GenerationandTransmissionofBitsandOutputoftheReceiverPacketization Decoder Input Function. MPEGpackets)inordertodecreasethepacketizationdelayandpossiblythedecodingtime ifthedecoderprocessesbitsassoonastheyenteritsbuer. P s B bitsarefeededtothedecoderonlyafterthewholepackethasbeenreceived.thus,bits bueringincomingpacketsandfeedingthedecoderbuerattheconstantrateb.received functioninthereceiverreconstructsthecbrmpegstream,asshowninfigure77,by BueringintheDepacketizationFunctionWeassumenowthatthedepacketization P P s s T 2T 3T Time exitingthecbrmpegencoderarebueredinthebuerofthepacketization(sender)and B C onegivenbyequation(33).thus,ifthedepacketizationfunctionreconstructsacbrmpeg depacketization(receiver)functionsforanoveralltime isdroppedandthelastbitofapicturemustnotbethelastofthepacketinwhichitis stream,theend-to-enddelayisincreased.theps=bincrementisnecessaryifassumption6 sent.infact,thesenderpacketizationfunctiondelaystherstbitofeachpacketbyps=b,as ThepacketizationdelayintroducedinthissystemcongurationisPs=Blargerthanthe Dp=Ps B+Ps underowswhen,forexample,thelastbitofapicturehappenstobetherstinapacket. showninfigure77;ifdecodingisnotdelayedbythesameamountoftime,thedecoderbuer 14 C (35)

Figure78:ArchitectureofaVideoconferencingSystemExploitingaCBRMPEGEncoder delay,thenetworkresynchronizationdelay,andtheexcessresynchronizationdelayertothe andanasynchronouspacketswitchednetwork. end-to-enddelaygivenbyequation(34): 5.7.2Multi-HopConguration Figure78showsthearchitectureofavideoconferencingsystemthatexploitsaCBRMPEG encoderandanasynchronouspacketswitchednetwork.thiscongurationaddsthequeueing Async Frame Encoder CBR=Sc+Ss+Dp+P+QM+Er+D+Pd Packetization Packetization Decoder (36) Grabber Adaptor wherepisthepropagationdelayonthelinksonthepathfromsourcetodestination,qm excessresynchronizationdelayintroducedbythereplaybuer(beingitinthepacketization isthemaximumqueueingdelayexperiencedbypacketsinthenetwork,er2[;q]isthe functionorinthedecoder)duetolackofsynchronizationbetweensenderandreceivernetwork delaydpisgivenbyequation(33)orequation(35)dependingonthebehaviorofthedepacketizationfunction(seesection5.7.1). dimensionofthereplaybuermustbe,atleast, Thisexpressionprovidesalsotheincrementrequiredinthedimensionofthedecoderbuer incaseitisusedtocompensatethenetworkdelayvariation. SincebitsareproducedattheconstantrateBandpacketsaresentatconstantpace,the interfaces(seesection3.4.1),qisthevariationofthequeueingdelay.thepacketization Thepacketizationfunctioncanimpactthestartupshapingdelay.Inmanypracticalcases 5.7.3PacketizationandStartupShapingDelay 2QB canbesucienttoproduceapacketworthofbits. timeps instantaneouslowbitproductionrate.ifthepacketizationfunctionisincludedintheclosed controlloopoftheencoder,thestartupshapingdelaycanbereducedoreveneliminated.the themeanbitproductionrateishigherthanthetargetrate;thusthestartupshapingdelay isessentialonlywhileencodingtherstmbsofapictureinordertocompensateapossible controlfunctiontakesintoaccountthatbitsarenotremovedfromtheencoderbuerfora B.Thus,eveniftheinitialbitproductionrateislow,theaverageoverthistimeinterval 15

RawVideoDed Raw=Fr Dedicated Link C+P+PdCS Raw=Sn+P+Sw+PdTDP Switching Circuit Sn=SAS Raw=Sn+LTf+Pd TimeDriven Priority n+(nr?1)tfpacketswitching Asynchronous Er+PdEr2[;Q] Async?Sh Raw=Fr Raw=Sn+Ps C+P+QM+ VBRMPEGDed VBR=CM+P+D+PdCS VBR=CM+Sw+P+ P+QM+Er+Pd P+Sw+D+Pd CS?TS VBR=CM+SCS n+ TDP?CxSc SSched n+ltf+d+pd VBR?I=CM+SAS SSched n2[;nt]async n2[;t] =CM+ D+PdC+P+QM+Er+ STS Async?TS QM+Er+D+Pd n+ps VBR=CM+Ps CBRMPEGDed D+Pd CBR=Sc+Ss+P+ P+D+Pd CS CBR=Sc+Ss+Sw+ Sn+Er+D+Pd CBR=Sc+Ss+LTf+ Er2[;Tc?Pf=B]Async Table2:SummaryoftheCongurationsConsideredinthisWork. Sn=SPack SPack SAS n2[;tc] n+sas np+qm+er+d+pd CBR=Sc+Ss+Dp+ Dp=Ps B+Ps C 16

keeptheend-to-enddelaybelow1ms.thisrequirementcomesfromthehumanhearing 6Summary Inthisworkweanalyzedtheend-to-enddelayofvideoconferencingoverpacketswitched networks.ourkeyndingsaresummarizedintable2.themaindesignobjectiveisto sensitivityfordelayslargerthan1msandtherequirementforlip-synchronization,i.e.,the needfortheaudioandvideotobesynchronized. (ii)networkand(iii)receiver.thecontrolofthebuersizescanhaveadverseconsequences: 1.Decreasingthebuersinsidethenetworkcanincreasethepacketlossinsidethenetwork Controllingthedelayalsorequirestocontroltheamountofbuersusedatthe(i)sender, Thus,whatwefoundandformulated,inthisstudy,aresometradeobetweentheperception qualityduetodelayandthereceivedpicturequalityduetolossandcompression. 2.LimitingtheMPEGencoderbuersizewilllimitthemaximumI-framesizeanddegrade Wefoundinterestingresultssomeofwhicharecounterintuitive.Thisinturnillustrates thecompressedmpegvideoquality. anddegradethequalityofthereceivedvideo. theimportanceofthissortofstudy. 1.Transmissionofrawvideodoesnotnecessarilyprovidetheshorterdelay.Thisisbecause 2.MPEGCBRencodingofaxedsceneintroduceslongdelay.ThisisbecausetheinterframecodingofP-framesrequiresfewbits,andtherefore,asingleI-framecanuseall thecbrcapacityallocatedtothegroupofpicture(gop).asaresult,thei-frame transmissionwilllasttheentiregopperiod. 3.Usingasynchronouspacketswitchingwithstatisticalmultiplexingischallenging.This ofthetransmissiontimeneededforlargenumberofbitsofhighdenitionpictures. 4.Ifthecapturecardandthedisplaybuerareusingthesamereferenceclockthedelay replaybueranddistortionofthevideoviewedbytheuseratthereceiver. largedelay,andsmallreplaybuerwouldresultinoccasionalunderow/overowofthe isbecausethedistributionofthedelayvariationorjitterinsidethenetworkcanbelarge reductionof1msandfor3framespersecondthedelayreductionis33ms. canbedecreasedbyonevideoframeperiod.for1framespersecondthismeansdelay andwithheavytail.thus,conservativedesignwithlargereplaybuerwouldresultin 5.MPEGVBRvideooverasynchronouspacketswitchednetworkrequireshighequivalent capacityoreectivebandwidth,whichcanbetoomuchtobewastedoverwidearea examplemotion-jpeg. links.therefore,insomecaseitmaybebettertouseonlyi-frames,i.e.,tousefor 17

end-to-enddelaybelowhumanperception,therearefourrequirements: 1.Thecapture(framegrabber)cardanddisplay(video)adaptershouldhaveacommon 2.Thepicturesshouldbesentrightafterthecompressioniscompletedasavariablebit Inconclusion,MPEG-basedvideoconferencingispossible,however,inordertokeepthe timereferenceclock. havethefollowingpropertiesforvbrmpeg: 1.Boundof25secondsonthejitter. 3.Thenetworkjittershouldbecontrolledwithawelldenedbound. Wealsoshowedthatwithtimedrivenpriorityandcomplexschedulingitispossibleto rate(vbr)stream. 2.Nolossevenifthelinkisfullyutilized. 3.Theend-to-enddelayisdominatedbythepropagationdelayplusL125seconds(L videoframeperiod.ifthesceneisslow,thisdelaycanbeaslargeasthedurationofa encodermustbedeployedwhichintroducesacodingshapingdelaylargerthanthe obtainedoveracircuitswitchednetwork.onacircuitswitchedconnectionacbr groupofpictures. dependsonthenumberofhops).thisdelayisshorterthanthedelaythatcanbe Acknowledgments WethankPrasoonTiwariforprovidinguswiththesoftwareMPEGencoderdvdencand PeterWesterinkforhiskindandusefulhelpinunderstanding,modifying,andoperatingthe encoder. AListofAchronymsandSymbols bmbmacroblocklevelratecontrolfactorofmacroblockmb. BtargetrateforMPEGCBRencoderorrateofcircuitswitchedconnectionsusedtosend Adimensionofthetokenbucketinaleakybucketshaper. CMmaximumcodingdelay. Ccapacityofphysicallinks. rawandencodedvideo. 18

CBRConstantBitRate. DCTDiscreteCosineTransform. ^cmb i;jdctcoecient(i;j)ofmbmb. DTidelayexperiencedinthenetworkbyframei i;jquantizeddctcoecient(i;j)ofmbmb. Ddecodingtimeordecoderdelay. DRMmaximumdelayexperiencedinthereplaybuerormaximumtransmissionsynchronizationdelay. DTmminimumdelayexperiencedbyaframeinthenetwork. DRidelayexperiencedinthereplaybuerortransmissionsynchronizationdelayexperienced byvideoframei. DTMmaximumdelayexperiencedbyaframeinthenetwork. ErExcessresynchronizationdelay Fnumberofbitsencodingaframe; Fmlowerboundonpicturedimension, GOPGroupOfPictures. Gglobaldistortionlevelthatischosenonapicturebypicturebasis FPdimensionofaP-frame. FIdimensionofanI-frame, Frnumberofbitsencodingarawpicture, MBMacroBlock. Mnumberofvideoframeperiodsinthetimecycle. Jnnetworkdelayjitter. Nnumberofvideoframesinagroupofpictures. Lnumberoftimeframesneededtotransferapacketfromsendertoreceiverwithtime drivenpriority. 19

Nsnumberoftimeframesinwhichbitshavebeenallocatedtosendaraworencodedvideo pmbmblocalactivityfactorofmbmb. NrnumberoftimeframesbetweentherstandthelastTFreservedforthetransmission Ppropagationdelay. ofapicture. framewithtimedrivenpriority. Pssizeofapacket. Qqueueingdelayinthenetwork; qfijpg i;jelement(i;j)ofquantizationmatrixfori-framesorp-frames. QoSQualityOfService. Qmbquantizationparameterofmacroblockmb. QMmaximumqueueingdelay. Qmminimumqueueingdelay, SCS nnetworkshapingdelayduetolackofsynchronizationbetweenapplicationandnetwork Snnetworkshapingdelay. Sccodingshapingdelay. STS nnetworkshapingdelayduetocircuitbandwidth. nnetworkshapingdelayduetotracshapingattheboundariesofasynchronouspacket interface. Swswitchingdelay. Ssstartupshapingdelay; switchednetworks. Tctimecycleduration. Tvideoframeperiod. Smsminimumstartupshapingdelay. TFTimeFrame. 11

VBVVideoBuerVerier. Tfdurationofatimeframe. BTheMPEGEncodingStandard Avideosequenceisdigitalizedbyconvertingintodigitalformpictures(orvideoframes)ata VssizeoftheVBV. Vbmaximumbacklogintheencoderbuer. framespersecondintheusand25framespersecondineurope. bitsusedtoencodeeachpixeldeterminesthepictureresolution.picturesaredescribedby threecomponentswhichareeithertheintensityofthered,green,andblue(rgbformat)or theyhadbeencaptured.forexample,goodqualityvideoisdigitalizedanddisplayedat3 xedpace.digitalvideoisreplayedbydisplayingpicturesatthesamexedpaceatwhich theintensityofluminanceandchrominance(yuvformat)ofthepixels. Eachpictureisdividedinpixels;thenumberofpixelsineachpictureandthenumberof broadcasting(e.g.,tvbroadcastingandvideoondemand). (1)storageandsubsequentretrievalanddisplay(e.g.,ondigitalvideodisks),and(2)oneway todigitallyencodeandcompressvideoandrelatedaudiomainlyfortwoapplicationelds: schemeshavebeenproposedwhichexploittheredundancyinside(spatialredundancy)and among(temporalredundancy)picturesinordertoreducethenumberofbitsneededtoencode avideosequence.themovingpicturesexpertgroupcodingstandard[11,25,8]specieshow Thiscodingformatrequiresahugeamountofbitstoencodesequences.Manycompression ferencingpurposes,eventhoughtheencodingalgorithmiscomputationallyintensiveandit ingandexplainingthecharacteristicswhichareparticularlyrelevanttothiswork. andoutputbitratesthroughthesamecodingprinciples. wasnotdesignedforreal-timeoperation.thempegstandardsupportsvariousresolutions ThisAppendixdescribesthebasicprinciplesofMPEGstandardwiththeaimofhighlight- Nevertheless,theMPEGstandardhasbeenproposedforbeingappliedalsoforvideocon- Compressionschemesarebasedonreductionoftemporalandspatialredundancyandquantizationofvaluesusedtocodeimages.Whileredundancyreductionisalosslessprocess, referenceone.thisintroducesstrongdependencyamongpicturesandpicturescodedby dierencecannotbedecodedanddisplayedpriortodecodingthereferenceimage.thisturns Temporalredundancyiseliminatedbycodingthedierencebetweenthepictureanda B.1GeneralPrinciples quantizationdegradesimagequality. 111

outtobeaproblemwhentransmittingacodedsequencebecausethelossof(partof)a referencepicturecausesallthereferencingimagestobecorrupted. temporalredundancyiseliminated: TheMPEGstandarddenesthreecategoriesofpicturesaccordingtowhetherspatialor Figure79:BlockDiagramoftheMPEGEncodingAlgorithmforI-Frames. I-frames(Intra-codedframes)arecodedbyeliminatingspatialredundancyonly.Hence Sequence Entropy of Raw DCT Quantization Images Encoding Stream P-frames(Predictive-codedframes)areencodedwithreferencetothepreviousimage. theyaredecodedbythemselves,withoutreferencetoanyotherpicture. Quantizer B-frames(Bidirectionallypredictive-codedframes)arecodedwithreferencetothepreviousandfollowingI-frameorP-frame.Thisallowssimilaritieswithafutureimage (possiblynotpresentwithinapreviouspicture)tobeexploitedtoreducetemporal redundancy.thiscodingyieldsverygoodcompression,butintroducesalargedelay rstimageisintra-codedandthefollowingpicturesinthesamegoparepredictive-coded21. thelowertheframerate,thelargerthedelayintroduced.sinceinthisworkweare becauseab-framecannotbecodeduntilthefuturereferencepicturehasbeencaptured; B.2Intra-codedPictures Whenavideosequenceisencoded,imagesaregatheredinGroupsofPictures(GOPs):the codingisnotexploitedandfurtherexplained. consideringinteractiveapplicationswhichrequirelowdelays,bidirectionalpredictive RawpictureswhicharetobeencodedwiththeMPEGstandardaredigitalizedinformatsimilartoYUV.Eachimageconsistsofaluminancecomponentandtwochrominancecomponents. Theformerhastwiceasmanypixelsasthelattertwocomponents. B.2.1DiscreteCosineTransformation basicstepsofintra-codingarebrieydescribed. Intra-framecodingyieldscompressionbyeliminatingspatialredundancy.InthisSectionthe 21Ingeneral,theGOPcontainsapatternofB-framesinterleavedbyP-frames,e.g.,IBBPBBPBBPBB. 112

inthefollowing)areidentied.thediscretecosinetransformation(dct)isperformed independentlyoneachblockandtherepresentationoftheblockinthefrequencydomainis obtained.thedcttakesasinput648-bitintegervaluessx;yrepresentingthevalueofeach pixelintheblockandproduces6412-bitintegersci;jwhicharetherepresentationoftheblock inthefrequencydomain[8]:ci;j=dct(sx;y)i;j;x;y2[;7] Oneachofthethreecomponentsofapicture8x8blocksofpixels(namedsimplyblocks block.theothercoecients,ac-coecients,representtheseparationoflinesineitheror bothdirections.e.g.,c7;isthehighestfrequencyappearingintheverticaldirectionand representstheclosestseparationofverticallinesintheblock.ci;iisproportionaltothe amountofsquaresintheblockbecausetheyarecharacterizedbythesamespatialfrequency inbothdirections.e.g.,thelargerthenumberofisolatedpixels,thehighestthevalueofc7;7. Thec;coecientiscalledDC-coecientandrepresentsthefundamentalcolorinthe compressionasthevaluestobeencodedarenotuniformlydistributed. groupedaccordingtothelevelofdetailintheblock.lowspatialfrequencycoecientscontaintheinformationontheoverallaspectoftheblock;highfrequencycoecientscontain theinformationconcerningthedetailsoftheblock.lowfrequencycoecientsusuallyhave largervaluesthanhighfrequencyones.theencodingstep(seesectionb.2.3)providesbetter Moreover,frequencydomaincoecientscanbecodedwithdierentprecisionaccording TheDCTprovidesenergycompactionbecausetheinformationcontainedinablockis tohowsensitivethehumaneyeistoeachofthem.lowfrequencyonesaremoreimportant indeterminingtheperceivedqualityofanimage,i.e.,theymustbeencodedwithminimal B.2.2Quantization EachDCTcoecientisquantizedwithadierentquantizerbecauseeachoneofthemweighs quality,andthustheycanbeencodedwithlessaccuracyandhighercompressionratio(coarser informationloss(nequantization).highfrequencycomponentsarelesscriticalfortheimage elementsoftheblockbythecorrespondingelementofthequantizationmatrix.thus,the higheristhevalueofquantizationelements,thecoarseristhequantization,i.e.,thegreater gerscalledquantizationmatrix.eachelementofthematrixindicatestheaggressivenessof quantizationonthecorrespondingdctcoecient:quantizationisperformedbydividingthe dierentlyontheimagequality.thequantizationisdrivenbyan8x8matrixof8-bitinte- isincludedatthebeginningofthempegvideostream. istheyieldedcompression.theresultofthisstepisa1-bitintegervalue[8]. Thequantizationmatrixischosenaccordingtothenatureofthescenetobeencodedand 113

c, c Thisstepproducesacompressedstreamofbitsthroughtwosubsteps[7,27]: B.2.3EntropyEncoding 1.Run-length.Thecoecientsobtainedbythequantizationarerun-lengthencodedwith Figure8:EncodingOrderforQuantizedDCTCoecients. asequenceofcouplesofsymbols: therstsymbol,isdividedintwoparts:,7 c 7, c Asaconsequence,foreachnon-zerocoecientanumberofbitsbetween8to18isused. thesecondsymbolisthecoecientvalueexpressedonsizebits;thevaluezero doesnothavetobecodedasitisrun-lengthencoded. (b)size(4bits)isthenumberofbits(betweenand1)usedtoencodethevalue (a)run-length(4bits)representsthenumberofcoecientswithvaluezero, ofthenextnonzerocoecient; 7,7 TheDC-coecientc;isencodedusingapredictivecodingtechnique,i.e.,thedierence oneisencoded.asthetwoquantizedcoecientsarebothrepresentedon1bits,the betweenthevalueofthedc-coecientofthecurrentblockandtheoneoftheprevious resultisencodedonamaximumof11bits. 2.Humanencoding.Theintermediatesymbolsequenceproducedbytherun-lengthencoderistransformedbyreplacingtherstsymbolineachcoupleofsymbolswitha similarbigvalues,whilehighfrequencyoneshavesmallervalues;theprobabilityofzero togetherallowsrun-lengthencodingtodeliverbettercompression. valuedcoecientsishigheramongthehighfrequencyonesandhavingthemgrouped beingconsideredinincreasingfrequencyorder.lowfrequencycoecientstypicallyhave TheAC-coecientsareencodedaccordingtothezig-zagordershowninFigure1,thus 114

ApplicationoftheMPEGencodingschemeoftenrequirespicturedimensiontobecontrolled: AsshowninSection4.1(seeFigure2)dimensionofencodedpicturesishighlyvariable. B.2.4ControllingI-frameDimension variablelengthcodeobtainedfromahumantablethathasbeenspeciedforeach e.g.,cbrmpegencoding(section5)andtransmissionofvbrmpegencodingthrough componentofthepicture. whenintra-codingapicture(i-framedimension)dependsonthefollowingfactors: packetswitchednetworkswithtimedrivenprioritysection4.3).thenumberofbitsobtained Resolutionofthedigitalizedimage:itisdenedforeachconferenceanditisnotusually Contentofthedigitalizedimage:iftheimagehasfewdetails,manyhighfrequencyDCT Quantizer:thehigherthequantizationstepsize,thesmallerthevaluesofthequantized changedontheyduringthesession. Varianceoftherstsymbolsproducedbyrun-lengthencoding:Humanencodingisas coecientshavesmallvalues.quantizationreducessmallvaluestozeroandrun-length encodingdelivershighcompression. Amongtheforegoingfactors,onlythequantizercanbechangedonpurposeanddynamicallytocontrolthecompressionratioandhenceI-framedimension.Thequantizationstepsize quantizationisregulatedseparatelyforeachcoecientinablock,thusallowinganetuning overawiderangeofcompressionratios(andimagequalities). ofzerovaluesyieldevenhighercompressionthankstorun-lengthencoding. symbolhasahighoccurrenceprobability. moreeectiveasthenumberofdierentrstsymbolsinthesequenceislowandeach coecients.smallerquantizedvaluesarehumanencodedwithfewerbits;sequences thatallowstheintendednumberofbitstobeproduced,whilekeepingimagequalityuniform. canbepossiblymodiedonablockbyblockbasis.sinceaquantizationmatrixisexploited, ofiterationstheintendeddimensionisobtainedforthei-frame. Theproblemissometimesaddressedwithreiterativeapproachesthatquantizeapicturemore thecoecientsarere-quantizedwithacoarser(ner)stepandre-encoded.afteranumber Ateachiterationthepictureisencoded;iftheyieldedamountofbitsistoobig(small),all thanoncewithdierentquantizersinordertoidentifythemostsuitablequantizationstepsize. OneofthemostdebatedtopicsincontrollingI-framesdimensionisthechoiceofaquantizer bitsthataregoingtobeproducedbycompressingani-framewithouthavingtoperformthe codingprocessuptotheend. beingsuitedtoreal-timeencoding.someheuristicscanbeusedtopredicttheamountof Thisapproachiscomputationallyhardanditdoesnotboundtheencodingtime,thusnot 115

1.TheDCTisperformedonthewholepicturebeforestartingquantizingandencoding 2.Duringquantizationsomeverysimpleandcomputationallylightalgorithmisrunin thecoecients.byaswiftanalysisoftheyieldedcoecientsthecompressionfactor providedbyhumanencodingcanberoughlyestimatedandasensiblequantization ordertogetanestimateoftheamountofbitsexpectedasoutcomeoftheencoding stepsizecanbechosen. 3.BlocksaregroupedinsetsandanequalshareofthetargetI-framedimensionisassigned toeachset.afteralltheblocksinasethavebeenprocessed,theyieldednumberofbits systemhasmorethanoneprocessor,morethanonequantizationprocesses,eachwith dierentstepsize,canberuninparallel. withalargerquantizationstepsizebeforeencodingtheoutcomeofquantization.ifthe step.ifthei-frameisexpectedtogrowtoolarge,thecoecientsarequantizedagain whoseblocksareencodedwithasmallamountofbits;morecomplexpartsarecharacterizedbyblocksrequiringahugeamountofbits.duetothepseudo-randomorder iscomparedagainsttheassignedshare.ifitissignicantlygrater(smaller)thanthe Blockgroupingmustbepseudo-randombecauseanimagecanhaveverysimpleparts withwhichblocksareencoded,theimagesendingcannotstartbeforethewholepicture hasbeenencoded. target,thecoecientsarere-quantizedwithalarger(smaller)quantizerandre-encoded. B.3Predictive-codedPictures Constraintsmustbeimposedonthevariabilityofthequantizationstepsizewithineachimage inordertopreventnon-uniformquality. Predictive-codingreducesbothspatialandtemporalredundancy.Picturesareencodedby Theproposedheuristicscanbeusedsingularlyorincompositiontogetbetterresults. thersttwoallowtemporalredundancytobereducedbeforegoingthroughtheverystepsof referencepictureinthefollowing.thissectiondescribesthebasicstepsofpredictive-coding; intra-coding. B.3.1MotionEstimation referencetothepreviousone(havingitbeenintra-orpredictive-coded)whichiscalledthe thetwochrominancecomponents).thereferencepictureissearchedforamb\similar"to (MBs)whichconsistof6correspondingblocks(4ontheluminancecomponentandoneon composedof4blocks;oneblockoneachofthechrominancecomponentscorrespondstothe 16x16-pixelsquareontheluminancecomponent.Motionestimationoperatesonmacroblocks Ontheluminancecomponentofthedigitalimage,16x16-pixelsquaresareidentied,each 116

theonebeingencoded;thepossiblyfoundmbiscalledapredictor.thedierencebetween thegivenmbandthepredictorisencodedasarepresentationoftheformer. theareaofthereferencepicturearoundthelocationcorrespondingtothembbeingencoded. Thealgorithmbehavioriscontrolledbytwoparameterswhichdetermineitsperformancein termsofrunningtimeandcontributiontotheoverallpicturecompression. 1.ThesearchrangeidentiestheareaaroundthecurrentMBpositioninwhichthereferencepictureissearchedforthepredictor.Thelargerthesearcharea,thehigherthe Analgorithm,notspeciedbytheMPEGstandard,isruninordertoidentifyapredictorin 2.ThesimilaritycriteriaaimsatchoosingaspredictortheMBwhichisgoingtoprovide complexthesimilaritycriteria,thebetterthecompressionobtained,buttheharderthe thehighestcompressionwhenthedierencefromtheactualmbisencoded.themore thesearch.thesearchrangecanbechangedonapicturebypicturebasis. probabilityofndingaverywellmatchingmb,thusyieldinggoodcompression.onthe computation. otherhand,thelargeristhesearcharea,thelongerittakestothealgorithmtocomplete wholeencodingprocess. B.3.2MotionCompensation WhenapredictorisfoundfortheMB,thedierencebetweeneachpixelandthecorresponding oneinthepredictoriscomputed.sincetwo8-bitintegersaresubtracted,theresultisa9-bit Motionestimationisthemostcomputationallyintensiveandtimeconsumingstepofthe block,i.e.,themotioncompensationstepisskipped. integer.ifambsimilarenoughisnotfound,eachblockofthembisencodedasani-frame sentationwhichallowsthepredictortobeidentied.motionisrepresentedthroughamotion previousmb. B.3.3DCT MBbeingencoded.Themotionvectorisencodedasdierencefromthemotionvectorofthe vector,i.e.,thebidimensionalosetofthepredictorfromthepositioninthepictureofthe ThecompleteencodingofamotioncompensatedMBencompassesalsoamotionrepre- predictor.ifnopredictorhasbeenfound,thedctisappliedonthepixelsofthembitself. thedctisperformedonthedierencesbetweenthepixelsofthegivenmbandthoseofthe TheDCTisperformedontheoutputofthepreviousstep.Ifapredictorhasbeenfound, 117

thembhasbeenfound)thembisnofurtherprocessedanditisrepresentedinthempeg besmallerthanthequantizationstepsize. thequantizermusthavenodeadzone,i.e.,therangeofvaluesthatarequantizedtozeromust andintra-codedmbs.themotioncompensatedmbslikelyhavesmallcoecientsandthus B.3.4Quantization TheresultoftheDCTisquantizedusingdierentquantizersformotioncompensatedMBs AllthecoecientsofmotioncompensatedMBsareencodedusingrun-lengthencodingand B.3.5EntropyEncoding streambyaspecial6-bitcode. IfthecoecientsofamotioncompensatedMBareallzero(i.e.,apredictoridenticalto Humanencoding(i.e.,theDC-coecientisnottreateddierently).Blocksofnonmotion compensatedmbsareencodedaccordingtotheintra-codingprocess. Thenumberofbitsobtainedwhenpredictive-codingapicture(P-framedimension)depends onthefollowingfactors: B.3.6ControllingP-frameDimension Dynamicsofscenes:ifscenesarestatic,subsequentpicturesdonotchangetoomuch, Searchrangeandsimilaritycriteriausedtondthepredictor.Bothparameterscouldbe Resolutionofimages:seeSectionB.2.4. motioncompensationishighlyeective,andp-framesaresmall. Quantizer:seeSectionB.2.4. usedtocontrolthecompressionratio,buttheyareemployedtooearlyinthecompression Varianceoftherstsymbolsproducedbyrun-lengthencoding:seesectionB.2.4. process.thus,ononeside,reiterationswithdierentvalueswouldbeimpracticabledue tothelargeamountofcomputationrequiredbymotionestimation.ontheotherside, ThesameactionsproposedinSectionB.2.4forcontrollingthesizeofI-frames,canbe isnottrivial. therelationshipbetweenchangingtheseparametersandthenumberofgeneratedbits motioncompensatedmbs,itispossibletoroughlyestimatetheamountofbitsthataregoing tobeproducedandconsequentlychoosethesearchrangeandthequantizerstobeused. bemotioncompensatedcanberoughlyestimatedbyestimatingtheoverallsimilarityofthe twoimages.havingsomeknowledgeorstatisticsaboutthenumberofbitsneededtoencode takenforp-frames.inaddition,beforeencodingap-frame,thepercentageofmbswhichcan 118

B.4SignaltoNoiseRatio reconstructedfromthempegstream.thesignaltonoiseratioisscaledaccordingtothe wheresiisthevalueoftheithpixelbeforeencoding,while^siisthevalueofthesamepixel Thelossduetotheencodingprocesscanbequantitativelymeasuredbythepeak-to-peak maximumvalueofthepixel(peak-to-peak)whichis255. SignaltoNoiseRatio(SNR)whichiscalculatedas SNR=1log12552 B.5PacketizationandStreamingPi(si?^si)2 (37) stampandadecodingtimestampwhichindicatethetimeatwhichtherstaccessunit22 streamtowhichthepacketbelongs.moreover,theheadercancontainapresentationtime TheMPEGstandard[11]denesanelementarystreamforcarryingeachmedia:video,audio, elementarystreamwiththempegstandardischeckedbyfeedingasystemtargetdecoder (STD)withthestreamandverifyingthatitsbuerdoesnotoveroworunderow. inthepacketisintendedtobepresentedanddecoded,respectively.thecomplianceofan ordata.asystemstreamisdesignedtocombineanumberofelementarystreams. Eachelementarystreamispacketized;eachpackethasaheaderidentifyingtheelementary fromthesystemstream. Thebueroverowsifthereisnotenoughspacetostoreawholepacketwhenitisextracted astheygettothedecoder.thebuerisemptiedbyretrievingawholeaccessunitatthe underowsifthecompleteaccessunitisnotyetinthebuerbeforeitsretrievalisscheduled. timespeciedbythedecodingtimestampcontainedinthepacketheader.thestdbuer Packets,possiblybelongingtodierentstreams,aregroupedinpacks,eachprecededby TheSTDbuerislledbythepacketsofthecorrespondingelementarystreamassoon atthedecoder. timeofarrivalofthelastbyteofthetimestampitself.itisusedtorecovertheencoderclock repeatedinsomeotherpacks.thesystemheadercarriesinformationthatisrelevantto apackheaderwhichcontainsinformationabouttherateofthestream.moreover,thepack elementarystream.thecurrentsizeofthestdbueroptionallyincludedinpacketheaders headercontainsatimestamp,calledthesystemclockreference,whichindicatestheintended thewholestreamandcontainsthemaximumsizeofthestdbuertobeusedwitheach mustbesmallerthanthemaximumvaluepresentinthesystemheader. 22Forexample,thetheaccessunitinavideostreamisapicture. Therstpackofthesystemstreamcontainsasystemheaderthatcanbeoptionally TheMPEG-2standardprovidestwokindsofsystemstream: 119

References 1.TheProgramStreamisdesignedforerror-freeenvironments,e.g.,storageandretrieval 2.TheTransportStreamisintendedforexploitationinlossyenvironments(e.g.,transmissionovercomputernetworks)andpacketshavexedlength(188bytes). ofvideoscenesfromdigitalvideodisks.itusesvariablelengthpackets. [1]M.Baldi,Y.Ofek,andB.Yener.Adaptiverealtimegroupmulticast.Submittedto [2]M.Butto,E.Cavallero,andA.Tonietti.Eectivenessofthe`leakybucket'policing [3]A.Demers,S.Keshav,andS.Shenker.Analysisandsimulationofafairqueueingalgorithm.ACMComputerCommunicationReview(SIGCOMM'89),pages3{12,1989. mechanisminatmnetworks.ieeejournalonselectedareasincommunications, IEEEINFOCOM'97. 9(3):355{342,April1991. [5]W.DingandB.Liu.RatecontrolofMPEGvideocodingandrecordingbyratequantizationmodeling.IEEETransactionsonCircuitsandSystemsforVideoTechnology, [4]W.DingandB.Liu.JointencoderandchannelratecontrolofVBRvideooverATM networks.invisualcommunicationsandimageprocessing,volume2668,pages392{ 47,1996. [7]B.Fufht.Asurveyofmultimediacompressiontechniquesandstandards.PartI:JPEG [6]J.Feng,K.-T.Lo,H.Mehrpour,andA.E.Karbowiak.Celllossconcealmentmethodfor standard.realtimeimaging,(1):49{67,1995. MPEGvideoinATMnetworks.InGLOBECOM'95,pages192{1924,1995. 6(1):12{2,February1996. [1]C.HorneandA.Puri.Videocodingwithadaptivequantizationandratecontrol.In [8]D.LeGall.MPEG:Avideocompressionstandardformultimediaapplications.CommunicationsoftheACM,34(4):47{58,April1991. [9]MovingPicturesExpertsGroup. [11]ISO/IEC.Informationtechnology-Codingofmovingpicturesandassociatedaudiofor JTC1/SC29/WG11/N4,April1993. VisualCommunicationsandImageProcessing,volume1818,pages798{86,1992. digitalstoragemediauptoabout1.5mbit/s.internationalorganizationforstandardization,1993. MPEG-2testmodel5. 12

[12]S.JungandJ.S.Meditch.AdaptivepredictionandsmoothingofMPEGvideoinATM [13]G.Karlsson.Asynchronoustransferofvideo.IEEECommunicationsMagazine,pages [14]C-S.Li,Y.Ofek,A.Segall,andK.Sohraby.Pseudo-isochronouscellswitchinginATM networks.inieeeinternationalconferenceoncommunications,pages832{836,1995. [15]C-SLi,Y.Ofek,andM.Yung.\Time-drivenpriority"owcontrolforreal-timeheterogeneousinternetworking.InIEEEINFOCOM'96,1996. 118{126,August1996. networks.inieeeinfocom'94,pages428{437,1994. [17]E.Linzer.ArobustMPEG-2ratecontrolalgorithm.Unpublishedtechnicalreport,IBM- [16]L.-J.Lin,A.Ortega,andC.-C.J.Kuo.Agradientbasedratecontrolalgorithmwith [18]W.LuoandM.ElZarki.AdaptivedatapartitioningforMPEG-2videotransmission applicationstompegvideo.pages392{395,1995. [19]P.PanchaandM.ElZarki.MPEGcodingforvariablebitratevideotransmission.IEEE T.J.WatsonResearchCenter,department924A. [2]D.Reininger,G.Ramamurthy,andD.Raychaudhuri.VBRMPEGvideocodingwithdynamicbandwidthrenegotiation.InIEEEInternationalConferenceonCommunications, pages1773{1777,june1995. overatmbasednetworks.ininternationalconferenceonimageprocessing,volume1, pages17{2,october1995. [21]I.E.G.RichardsonandM.J.Riley.UsageparametercontrolcelllosseectsMPEG CommunicationsMagazine,pages54{66,May1994. [22]R.M.Rodriguez-Dagnino,M.R.K.Khansari,andA.Leno-Garcia.Predictionofbitrate [23]M.Simon,P.Villegas,J.Caballero,andM.Roser.Ageneralapproachtooutputrate 9(3):35{313,April1991. 1995. sequancesofencodedvideosignals.ieeejournalonselectedareasincommunications, video.inieeeinternationalconferenceoncommunications,pages97{974,june [24]S.SinghandS-S.Chan.Amulti-levelapproachtothetransportofMPEG-codedvideo controlinvideocoding.invisualcommunicationsandimageprocessing,volume193, pages246{254,1993. overatmandsomeexperiments.inieeeglobecom'95,pages192{1924,1995. 121

[27]G.K.Wallace.TheJPEGstillpicturescompressionstandard.Communicationsofthe [25]R.SteinmetzandK.Nahrstedt.Multimedia:computing,communications&applications. [26]A.SultanandH.A.Latchman.AdaptivequantizationschemeforMPEGvideocoders volume2668,pages181{188,1996. PrenticeHall,UpperSaddleRiver,NJ7458,1995. [28]L.Wang.RatecontrolforMPEGvideocoding.InVisualCommunicationsandImage Processing,volume251,pages53{63,1995. ACM,34(4):3{44,April1991. basedonhvs(humanvisualsystem.invisualcommunicationsandimageprocessing, 122