Turbo Coding and MAP decoding - Part 1

Transcription

1 Intitive Gide to Principles of Commnications Trbo Coding and MAP decoding - Part Baye s Theorem of conditional probabilities Let s first state the theorem of conditional probability, also called the Baye s theorem. P( A and B P( A P( B givena Which we can write in more formal terminology as P( A, B P( A P( B A A where P(B A is referred to as the probability of event B given that A has already occrred. If event A always occrs with event B, then we can write the following expression for the absolte probability of event A. P( A P( A, B B B If events A and B are independent from each other then (A degenerates to P( A, B P( A P ( A, B P( A P( B C The relationship C is very important and we will se it heavily in the explanation of Trbo decoding in this chapter.

2 If there are three independent events A, B and C, then the Baye s rle becomes P( A, B C P( A C P( B A, C D A-priori and a-posteriori probabilities Here is Bayes theorem again. P( A, B P( AB P( B Pr obability of a posteriori a priori both A and B probability of probability of event A event B P( B A P( A a posteriori a priori probability of probability of event B event A E The probability of event A conditioned on event B, is given by the probability of A given B times the probability of event a. The probability of A, or P(A is the base probability of even A and is called the a-priori probability. The term P(A,B the conditional probability is called the a-posteriori probability or APP. One is independent probability, the other depends on some event occrring. We will be sing the acronym APP a lot, so mae sre yo remember that is the same a-posteriori probability. In other words, the APP of an event is a fnction of an another event also occrring at the same time. We can write (E as P( A, B P( AB APP P( B A P( A PB ( F This says that we can determine the APP of an event by taing the conditional probability of that event divided by it s a-priori probability. What these mean is best explained by the following two qotes. In epistemological terms A priori and a posteriori refer primarily to how, or on what basis, a proposition might be nown. In general terms, a proposition is nowable a priori if it is nowable independently of experience, while a proposition nowable a posteriori is nowable on the basis of experience. The distinction between a priori and a posteriori nowledge ths broadly corresponds to the distinction between empirical and nonempirical nowledge. [] Bt how do we decide when we have gathered enogh data to jstify modifying or prediction of the probabilities? That is one of the essential problems of decision theory. How do we mae the transition from a priori statistics to a posteriori probability? [3]

3 3 The MAP algorithm helps s mae the transition from a-priori nowledge to nowledge based on received data. Strctre of a Trbo Code According to Shannon, the ltimate code wold be one where a message is sent infinite times, each time shffled randomly. The receiver has an infinite versions of the message albeit corrpted randomly. From these copies, the decoder wold be able to decode with near error-free probability the message sent. This is the theory of an ltimate code, the one that can correct all errors for a virtally signal. Trbo code is a step in that direction. Bt it trns ot that for an acceptable performance we do not really need to send the information infinite nmber of times, jst two or three times provides pretty decent reslts for or earthly channels. In Trbo codes, particlarly the parallel strctre, Recrsive systematic convoltional (RSC codes woring in parallel are sed to create the random versions of the message. The parallel strctre ses two or more RSC codes, each with a different interleaver. The prpose of the interleaver is to offer each encoder an ncorrelated or a random version of the information, reslting in parity bits from each RSC that are independent. How independent these parity bits are, is essentially a fnction of the type and length/depth of the interleaver. The design of interleaver in itself is a science. In a typical Viterbi code, the messages are decoded in blocs of only abot bits or so, where as in Trbo coding the blocs are on the order of 6K bits long. The reason for this length is to effectively randomize the seqence going to the second encoder. The longer the bloc length, the better is its correlation with the message from the first encoder, i.e. the correlation is low. On the receiving side, there are same nmber of decoders as on the encoder side, each woring on the same information and an independent set of parity bit. This type of strctre is called Parallel Concatenated Convoltional Code or PCCC. The convoltional codes sed in trbo codes sally have small constraint length. Where a longer constraint length is an advantage in stand-alone convoltional codes, it does not lead to better performance in TC and increases comptation complexity and delay. The codes in PCCC mst be RSC. The RSC property allows the se of systematic bit as a standard to which the independent parity bits from the different coders are sed to assess its reliability. The decoding most often applied is an iterative form of decoding. When we have two sch codes, the signal prodced is rate /3. If there are three encoders, then the rate is ¼ and so on. Usally two encoders are enogh as increasing the nmber of encoders redces bandwidth efficiency and does not by proportionate increase in performance.

4 4 EC p y, s y, n EC ECn p y, n p y, Figre A rate /(n+ Parallel Concatenated Convoltional Code (PCC Trbo Code Trbo codes also come as Serial Concatenated Convoltional Code or SCCC. The SCCC codes appear to have better performance at higher SRs. Where the PCCC codes reqire both constitent codes to be RSC, in SCCC, only the inner code mst be RSC. PCCC codes also seem to have a flattening of performance arond -6 which is less evident in SCCC. The SCCC constitent code rates can also be different as shown below. The oter code can even be a bloc code. In general the PCCC is a special form of SCCC. We can even thin of concatenation of RS/Convoltional codes, sed in line-of-sight lins as a form of SCCC. A Trbo SCCC may loo lie the figre below with different rate constitent codes. EC EC Rate / Rate /3 Figre Serially concatenated constitent coding (SCCC Then there are also hybrid versions that se both PCCC and SCCC sch as shown in figre below. EC EC EC3 Figre 3 Hybrid Trbo Codes There is an another form called Trbo Prodct Code or TPC. This form has a very different strctre from the PCCC or SCCC. TPC se bloc codes instead of convoltional codes. Two different bloc codes (sally Hamming codes are concatenated serially withot an

5 5 interleaver in between. Since the two codes are independent and operate in rows and colmns, this alone offers enogh randomization that no interleaver is reqired. TPC codes, lie PCCC also perform well in low SR and can be formed by concatenating any type of bloc codes. Typical coding method is to array the coded data in rows and then the second code ses the colmns of the new data for its coding. The following shows a TPC code created from a (7x5 and a (8x4 Hamming code. The 8x4 code first codes the 4 info bits into 8, by adding 4 p party bits. These are arrayed in five rows. Then the 7x5 code wors on these in colmns and creates (in this case, both codes are systematic new parity bits p for each colmn. The net code rate is 5/4 ~.33. The decoding is done along rows and then colmns. Hamming (5x8 (8x5 Hamming (7x8 (8x4 (7x5 i i i i p p p p i i i i p p p p i i i i p p p p i i i i p p p p i i i i p p p p p p p p p p p p p p p p p p p p Figre 4 Trbo Prodct codes What maes all these codes Trbo is not their strctre bt a form of feedbac iterative decoding. If the strctre of a SCCC does not se the iterative coding then it wold be jst a plain old concatenated code, not a trbo code. Maximm a-posteriori Probability (MAP decoding algorithm Trbo codes are decoded sing a method called the Maximm Lielihood Detection or MLD. Filtered signal is fed to the decoders, and the decoders wor on the signal amplitde to otpt a soft decision The a priori probabilities of the inpt symbols is sed, and a soft otpt indicating the reliability of the decision (amonting to a sggestion by decoder to decoder is calclated which is then iterated between the two decoders. The form of MLD decoding sed by trbo codes is called the Maximm a-posteriori Probability or MAP. In commnications, this algorithm was first identified in BCJR. And that is how it is nown for Trbo applications. The MAP algorithm is related to many other algorithms, sch as Hidden Marov Model, HMM which is sed in voice recognition, genomics and msic processing. Other similar algorithms are Bam-Welch algorithm, Expectation maximization, Forward-Bacward algorithm, and more. MAP is a complex algorithm, hard to nderstand and hard to explain.

6 6 In addition to MAP algorithm, another algorithm called SOVA, based on Viterbi decoding is also sed. SOVA ses Viterbi decoding method bt with soft otpts instead of hard. SOVA maximizes the probability of the seqence, whereas MAP maximizes the bit probabilities at each time, even if that maes the seqence not-legal. MAP prodces near optimal decoding. In trbo codes, the MAP algorithm is sed iteratively to improve performance. It is lie the qestions game, where each previos gess helps to improve yor nowledge of the hidden information. The nmber of iteration is often preset as in qestions. More iteration are done when the SR is low, when SR is high, lesser nmber of iterations are reqired since the reslts converge qicly. Doing iteration maybe a waste if signal qality is good. Instead of maing a decision ad-hoc, the algorithm is often pre-set with nmber of iterations. On the average, seven iterations give adeqate reslts and no more are ever reqired. These nmbers have relationship to the Central Limit Theorem. DEC DEC Figre 5 Iterative decoding in MAP algorithm Althogh sed together, the terms MAP and iterative decoding are separate concepts. MAP algorithm refers to specific math. The iterative process on the other hand can be applied to any type of coding inclding bloc coding which is not trellis based and may not se MAP algorithm. I am going to concentrate only on PCCC decoding sing iterative MAP algorithm In part, we will go throgh a step-by-step example. In this part, we will cover the theory of MAP algorithm. We are going to describe MAP decoding sing a Trbo code in shown Figre 5 with two RSC encoders. Each RSC has two memory registers so the trellis has for states with constraint length eqal to 3.

7 7, s c, p c,s c Transmitted symbol i t x ( x n i r Recevied symbol y y y,s, p,s,s y, p y DEC +,s, p y y DEC + +, p c Figre 6 A rate /3 PCCC Trbo code in a 8PSK channel The rate /3 code shown here has two identical RSC convoltional codes. The coding trellis for each is given by the figre 7. The ble lines show transitions in response to a and red lines in response to a. The notation /, the first is the inpt bit, the next are code bits. Of these, the first is what we called the systematic bit, and as yo can see it is the same as the inpt bit. The second bit is the parity bit. Each code ses the same trellis for encoding. The labels along the branches can be read as / c c. Since this a RSC, the first code bit is same as the information bit or c. The info bits are called. The coded bits are referred to by the vector c. Then the coded bits are transformed to an analog symbol x and transmitted. On the receive side, a noisy version of x is received. By looing at how far the received symbol is from the decision regions, a metric of confidence is added to each of the three bits in the symbol. Often Gray coding is sed, which means that not all bits in the symbol have same level of confidence for decoding prposes. There are special algorithms for mapping the symbols (one received voltage vale, to M soft-decisions, with M being the M in M-PSK. Let s assme that after the mapping and creating of soft-metrics, the vector y is received. One pair of these decoded soft-bits are sent to the first decoder and another set, sing a de-interleaved version of the systematic bit and the second parity bit are sent to the second decoder. Each decoder wors only on these bits of information and pass their confidence scores to each other ntil both agree within a certain threshold. Then the process restarts with next symbol in a seqence or bloc consisting of symbols (or bits. Definitions is the frame size of transmitted symbols. So for a M-PSK, there wold be 3 bits transmitted per frame, of these /3 will be the information bit. In a typical Trbo code, there may be as many as, smbols in a frame.

8 8 The seqence of information bits is given by. The first encoder gets = (,, 3, and the second encoder gets a preset reordering of this same seqence. For example, we may pic this mapping fnction.,, 3, 4, 5, 6, , 4,,, 5, 6, 7..., s, p The encoder mapping is be given by the vector c. The c ( c, c are two bits prodced, s, p by the first encoder, and c ( c, c are the two bits prodced by the second encoder. The information bit to code bits mapping is done a trellis sch as the one described in Table I. Table I Mapping of information bits to code bits s s s s s3 s3 s4 s4 Figre 7 the trellis diagram of the rate / code RSC code The symbol described by vector x (3 bits per vector is sent for each time i. Let s call this vector x i. There wold of these symbols transmitted. x,, x,,

9 9 y y y y are the soft mapped bits. The goal is to tae these and mae a gess,,, ( s, p, p abot the transmitted x vector and hence code bits which intrn decode, the information bit. Of this three soft bits, each decoder gets jst two of these vales. The first decoder, s, p, s, p gets y ( y, y and the second decoder gets y ( y, y which are its respective received data. Each decoder wors with jst two vales. The second decoder however gets a reordered version of the systematic bit, so it is getting only one bit of independent information. Log-lielihood Ratio (LLR Let s tae the information bit, a binary variable,, where is its vale at time. Its Loglielihood Ratio (LLR is defined as the natral log of its base probabilities. P ( L( ln P( (. If has two vales, + and - volts representing and bit and since these are eqally liely, as they are in most commnication system, then this ratio is eqal to zero. This metric is sed in most error correction coding and is called the log lielihood ratio or LLR. This is a sensitive metric, qite a bit better than a linear metric. Logs mae it easy to deal with very small and very large nmbers, as yo well now. ow note what happens to this metric, if the the binary variable is not eqally liely, as happens in trellis decoding. From elementary probability theory, we now that sm of all probabilities of an event add to. So we can write that the probability of = + as mins the probability of = -. P( P( ow sing eqation (., rewrite the expression of LLR from (. as. L( P ( ln P ( (. is plotted in Figre 8 as a fnction of the probability of one of the events. Let s say we are given L( =.. Then the probability that = + is eqal to.73

10 LLR Trbo Coding and MAP Decoding Probability, = + Figre 8 The range of LLR is from to + and is a direct indication of the bit. As we can see, if LLR, a non-dimensional metric, is positive, it is a pretty good indicator of the sign of the bit. This is an even better indicator than the Eclidean distance since its range is so large. LLR a very sefl parameter and we will see how it is sed in Trbo decoding. In (. formlate the LLR of a decoded bit, at time, conditioned on the received signal, a seqence of bits. The lower index of y means it starts at time and the pper index means the ending point at. P( y L ( ln P ( y (. We can reformlate the nmerator and denominator sing Baye s rle C. P( y, / P( y P( y, L ( ln ln P( y, / P( y P( y, (.3 This formlation of the LLR incldes joint probabilities between the received bits and the information bit, the nmerator of which can be written as (.4. For RSC trellis, each path is niqely specified by any pair of these:. the start state s,. the ending state s, 3. The inpt bit. If we now any two of these pieces of information, we can identify the correct path withot error. In this eqation, we only now the whole seqence y, we do not now, nor do we have any other of the piece of information. Bt what we do now is that saying a = is same as saying that the correct path is one of the for possible paths shown in Fig. 8. So the joint probability of ( y, (having received the bit seqence, the probability of at time = is the same as replacing the information bit with ending and starting states. These formlations are eqivalent.

11 P(, y P( s, s, y (.4 s s s s s3 s3 s4 s4 Figre 9 Trellis transition possible at any time in response to a + and -. We have changed the left hand side of the eqation by replacing = + with two states s and s. The s is the starting state and s is the ending state for all allowable transitions for which the = +. There are for valid transitions related to decoding a +. Assme that there is a probability associated with each of these for transitions. This says that the probability of maing a decoding decision in favor of a + is the sm of the probability of all for of these possible paths. ow plg Error! Reference sorce not fond. into (.3 to get the log lielihood ratio of as P s s y P( y, P y P s s y (,, L ( ln (, (,, (.5 ow we need to mae one more conceptal leap. The probability of deciding which road, i.e. the + road or the - road taen is a fnction of where the path started and where it will end which are sally given as bondary conditions. If we split the whole received seqence into manageable parts, it may help s identify the starting or ending states. We incorporate these ideas into (.5 to get, y y, y, y y, y, y p f We tae the bit seqence and separate it into three pieces, from to -, then the th point, and then from + to. We adopt a slightly easier terminology in (.6. P( s, s, y P( s, s, y, y, y (.6 p f

12 y p y y f The past seqence, the part that came before the crrent data point. The crrent data point The seqence points that come after the crrent point. Rewrite sing (.6 sing Baye s rle of joint probabilities (D. P s, s, y P s s y ( (,, yp,, yf P s s ( yf,, yp, y P( s, s, yp, y (.7 This loos complicated bt we are jst applying Bayes rle to simplify (.6 and to breadown the terms in terms of past, present and ftre parts of the seqence. So that whenever we are maing a decision abot a bit + or a -, a cmlative metric will tae into accont the starting and the ending point of the seqence. The starting and ending points of a convoltional seqence are nown and we se this information to winnow ot the lielier seqences. The term yf is the ftre seqence and we mae an assmption that it is independent of the past and only depends on the present state s. We can remove these dependencies to simplify (.7. P s, s, y P s s y P s s y ( ( y f,, yp, (,, yp, P s P s y ( y f (, s, y p, (.8 ow apply Baye s rle to the last term in (.8, to get P s s y y P s y s y P s y (,, p, (,, p (, P s, s, y P y s P s y s y P s y (.9 ( ( f (,, p (, p ow define a few new terms.

13 3 ( s Ps (, yp ( s P( y s (. f ( s, s P( s, yc s, yp The terms,, on the right are the metrics we will compte in MAP decoding. ow the nmerator term of (.5 becomes. P( s, s, y ( s ( s ( s, s (. We can plg this into (.5 to get the LLR eqation for MAP algorithm. L( ( s ( s ( s, s ( s ( s ( s, s (. ( s This first term in (. is called the Forward metric. ( s is called the Bacward metric. ( s, s is called Transition metric. The MAP algorithm allows s to compte these metrics. Decision is made at each time, abot the transmitted bit. Unlie Viterbi decoding where decision is made in favor if the lieliest seqence by carrying forward the sm of metrics, here the decision is made for the lieliest bit. The MAP algorithm is also nown as Forward-Bacward Algorithm becase we are assessing probabilities in both forward and bacward time. How to calclate the Forward metrics We will start with the forward metric at time. ( s Ps (, s, y p, y All states Which is also eqal to the probabilities at the previos state s times the transition probability to the crrent state s.

14 4 ( s P( s, s, yp, y ( s, s. ( s (.3 All states All s The initial condition is that since we always start at state, ( s if s otherwise Example: (. ( ( ( ( (.63 s s s3 ( ( - + s4 Figre - Compting Forward metrics The left most vale at the top is the starting point, so it has a vale of (, the starting vale of is = at all the remaining three states. The ending forward metric at t = is ( s ( s, s. ( s All s ( ( otice that these nmbers are larger than, which means, they are not probabilities bt are called Metrics. It also means that since a typical Trbo code frame is thosands of trellis sections long that mltiplication of these nmbers may reslt in nmerical overflow. For that reason, both and are normalized. Bacward Metric ( s

15 5 Same as the initial vales of (, we assme that since trellis always ends at state, all probabilities at this point are eqal to or. ( and zero elsewhere. This probability is eqal to the prodct of all transition probabilities times the probability at the last state woring bacwards. s s (.4 ( ( ( s, s All s Compare this with forward metric eqation. ( s. ( s ( s, s Alls s.855 ( (. s.5 (.9 s3 s ( ( ( (3 (4 Figre - Compting Bacward metrics ( s ( s ( s, s Alls ( ( Yo can see these vales at time + for state and. These are then normalized.

16 6 merical isses In order to prevent data overflow that can happen in the very large trellis of a Trbo codes,, (s and ( s need to be normalized. For forward metrics, we normalized them by their sm. ( s ( s ( s s (.5 The reverse metric is similalry normalized by the same qantity as above. It s formlation loos different bt it is the same thing as the term that normalized the forward metric. (Change index to instead of -. ( s s s ( s ( s ( s, s. (.6 How to calclate transition probabilities Compting transition metrics trns ot to be the hardest part. To restate the definition of the transition metric from (. ( s, s P( s, yc s, yp The transition itself does not depend on the past, so we can remove the dependency on the past and then apply the Baye s theorem. ( s, s P( s, y s, yp P( s, y s P y s s P s s (,. ( (.7 (, (,. ( s s P y s s P (.8 There are two terms in this final definition of the transition metric. Let s tae the last one first. Here P( is a-priori probability of the inpt bit. Let s tae its log lielihood. P( P( L ( log log P( P( Or by taing this expression to power of e, we get

17 7 L ( P ( e P ( From here, some clever algebra gives s P e P L ( ( ( ( e e e e e L ( L ( L ( L ( / L ( e L ( / ow we get the general expression for both, + and -. P e L( / L( / e (.9 L( ( e The term nderlined is a common factor and designated by A e e L ( / L ( The a-priori probability is now given by L ( / (. P( A e ow for the second term in transition metric expression of (.8, P( y s, s can be written as n (, ( ( l l l P y s s P y c P y c For rate ½, we can write this simply as, P( y x y c y c, s, p This is jst a correlation of the inpt signal vales with the trellis vales, c. The received signal vales y are assmed to be corrpted by an AWG process. The probability P( y c is given in a Gassian channel by l l

18 8 P( yl cl exp ( y l c l (. Apply (. to (.8, to get,,,, i p i p y s c s y c ( / q P y EXP i n n Expanding this we get, i p pi, y c, s q,, s, s, s i, p i, p ( y ( c y c q y c EXP EXP i i n n n n The sqare of the two c (always eqal to or - vales is eqal to, since sqare of + and - is the same. This gives the following eqation, s, s q i, p i, p y c y c B EXP (. n i n Where B EXP ip, y, s q ( y n i n. ow we can write the fll eqation for transition metric by combining (. and (.9.

19 9 (, (,. ( s s P y s s P.,, i, p i, p y s c s q y c Lc ( / B exp A e i n n (.3 A and B are constants and are not important becase when we pt this expression in the LLR form, they will cancel ot. The index q = for rate ½ code we are sing for example. ow we define, E p c n coderate Eb / Eb / (.4 Where p is inverse of code rate, i.e. eqal to for code rate = / E b Ec rate. (.5 where E b code rate E b. is ratio of the n-coded bit energy to noise PSD, Ec is coded bit energy, and Ec= p Sbstitte for n E / b into (.3 we have, s, q i p i 4 E / b y c y c A B exp L( c c p i

20 A B 4 E / y c y c L c c,, exp q b s i p i ( p i 4 E / 4 E /,, exp ( exp q b s b i p i A B L c c y c y x p i p With 4 E / p b Lc 4. Es / A B L c c Lc y c Lc y c (.6,, exp ( exp q s i p i i where we define a new partial transition metric, the nderlined part of (.6. e, (, exp q i p i s s Lc y c. i In smmary the LLR of the information bit at time is, L( log ~ ~ ~ ( s ( s, s ( s ~ ( s ( s, s ( s (.7 ~ ( s ( s, s ~ ( s ~, ( s ( s, s s s s ~ if s ( s (.8 otherwise

21 ~ ( s ( s, s ~ s ( s ~, ( s ( s, s s s ~ if s ( s (.9 otherwise,, (, exp ( exp q e s i p i s s L c c Lc y c Lc y c (.3 i Sbstitte (.3 into (.7, to get log (.3 ~ ~ ~ e ( s ( s exp L ( c ~ e ( s ( s exp L ( c c c Lc y Lc y, s, s c c exp exp q i q i Lc Lc y y i, p i, p c i c i If q =, the eqation become log ( s ( s exp L ( c c Lc y c exp Lc y c ( s ( s exp L ( c c Lc y c exp Lc y c e, s, p e, s, p L( and Lc y, s The factor ½ and can be plled ot becase both nmerator and denominator are constant. c cancel. We get, ( s ( s ( s, s e (, s ( log e L L c Lc y Where ( s ( s ( s, s

22 q e i, p i ( s, s exp Lc y c i This eqation can be broen into three distinct nmbers. They are L _ apriori L _ channel L _ extrinsic (.3 Lc is the a-priori information abot, ( Lc is the channel vale calclated from the nowledge of the SR and the received signal. The third term is called the a-posteriori term, also called the extrinsic L vale. L _ extrinsic log ~ ~ ~ e ( s ( s ( s, s ~ e ( s ( s ( s, s, s y (.33 Dring each iteration the decoder prodces the extrinsic vale sing the first two nmbers as inpt. The extrinsic vale it prodces becomes the inpt to the next decoder. L ( c, s e e Lc y L ( c L ( c, And eventally a decision is made abot the bit in qestion by looing at the sign of the L vale. ~ sign{l ( c }, The process can contine ntil the extrinsic vales stop changing with a preset threshold. Or the algorithm can allow jst a fixed nmber of iterations., p y, p y,s y DEC Extrinsic L vale from to,s y DEC Extrinsic L vale from to Figre Iterative natre of Trbo decoding.

23 3 So that s abot it. I hope it was helpfl. The next ttorial goes throgh the MAP algorithm in a step-by-step example. Please contact me if yo find errors. Charan Langton Copyright, 6, 7 Charan Langton All Rights reserved

24 4 References [] Trbo code in IS- Division Mltiple-Access Commnications Under Fading By Jian Qi. Masters Thesis, Wichita State University, Fall 999. Comment: Available on the Internet. I sed Ms. Qi s terminology and heavily relied on this wor. Its an excellent reference. [] "A Priori and A Posteriori" - an article by Jason Baehr in the Internet Encyclopedia of Philosophy. [3] [4] ot complete