Saddlepoint Approximation of the Cost-Constrained Random Coding Error Probability

Size: px

Start display at page:

Download "Saddlepoint Approximation of the Cost-Constrained Random Coding Error Probability"

Clara Maryann Booth
5 years ago
Views:

1 Saddlepoit Approximatio of the Cost-Costraied Radom Codig Error Probability Josep Fot-Segura Uiversitat Pompeu Fabra Alfoso Martiez Uiversitat Pompeu Fabra Albert Guillé i Fàbregas ICREA ad Uiversitat Pompeu Fabra Uiversity of Cambridge guille@ieee.org Abstract Saddlepoit approximatios to the pairwise error probability ad to the radom codig uio boud are derived for the cost-costraied radom codig esemble. For the special case of the AGN chael, a alterative expressio to approximate the Shao boud for optimal spherical codes is foud. I. INTRODUCTION Coceived by Shao, the idea of radom codig has bee oe of the mai proof techiques i iformatio theory. By geeratig a code esemble whose codewords are i.i.d. distributed, the average performace of such esemble guaratees the existece of a code with vaishig error probability as log as the code rate is smaller tha the mutual iformatio. May applicatios require codewords to satisfy a cost costrait, such as a maximum trasmitted power, Ch. 7. Uder cost-costraied radom codig, codewords are geerated accordig to a cost coditioed distributio. This is similar to the idea of costat-compositio codes where all codewords have the same empirical distributio 3. Both costcostraied ad costat-compositio esembles may lead to performace gais over the i.i.d. esemble 4 6. I this work, we study saddlepoit approximatios to the cost-costraied radom codig error probability. For a fixed codig rate R below the chael capacity, the cost-costraied radom codig error expoet E(R) provides a first approximatio of the error probability as e E(R), where is the code legth. Saddlepoit approximatios 7 aim at fidig a more refied approximatio of the form α e E(R), where α is a subexpoetial factor. Such characterizatio has bee recetly addressed by 8 for the ucostraied case. For the cost-costraied esemble, we derive a estimate of α valid uder some o-lattice coditios, ad for rates above the critical rate. e later cosider the particular case of shell iput AGN chael, ad revisit the Shao bouds o the performace of optimal spherical codes. II. COST-CONSTRAINED RANDOM CODING e cosider a radom codig esemble i which codewords are draw from a cost-costraied distributio P (x) give by P (x) = Q(x i ) {x D }, () i= This work has bee fuded i part by the Europea Research Coucil uder grat 754, ad by the Spaish Miistry of Ecoomy ad Competitiveess uder grats FJCI ad TEC C3--R. where is a ormalizig factor, Q(x) is a give distributio, { } is the idicator fuctio, ad D is a cost-costrait set D = {x : δ c (x) δ }. () I (), c (x) = i= c(x i) is a cost fuctio satisfyig E Q c (X) =, ad we assume that δ ad δ are costats idepedet of. Sice () is a probability distributio, it follows that = P Q X D. For every message m, equiprobably distributed over the set {,..., M}, a codeword x m is idepedetly geerated accordig to (). A codeword x m is trasmitted through a memoryless chael characterized by the trasitio probability (y x m ). For a give codebook, let ɛ (x,..., x M ) be the probability of havig a error with maximum likelihood (ML) decodig. The, radom codig argumets prove the existece of a code with a vaishig error probability as good as, at least, the average error probability over the esemble, deoted as ɛ (M) = E P ɛ (X,..., X M ). III. SADDLEPOINT APPROXIMATIONS Usig the uio boud, the radom codig error probability ɛ (M) is upper bouded by ɛ (M) rcu(m), where the radom codig uio (RCU) boud is give by rcu(m) = E P mi {, (M )pep(x, Y )}, (3) ad the pairwise error probability pep(x, y) is the probability that a idepedetly geerated codeword x has a larger decodig metric tha the trasmitted codeword x, i.e., pep(x, y) = P P (y X) (y x). (4) e first provide a saddlepoit approximatio to the pairwise error probability (4), ad the use this approximatio to fid a secod saddlepoit approximatio to the RCU boud (3). A. Pairwise Error Probability The pairwise error probability (4) is the probability of the error evet E (x, y) give by E (x, y) = {x : log (y x) log (y x)}, (5) for fixed x ad y. The, { pep(x, y) = E P X E (x, y) } (6) = { E Q X D E (x, y) }, (7)

2 where i (7) we have used the cost-costraied distributio () (). e defie the radom variables Z(x, y) = log (y X) log (y x) ad V = c (X). For a strogly o-lattice two-dimesioal radom variable (Z(x, y), V ), ad usig the defiitios of D ad E (x, y), equatio (7) ca be writte as pep(x, y) = δ dz dv p(z), (8) δ where we have defied the colum vector z = (z, v) T for coveiece, ad p(z) is the jowiit probability desity fuctio of Z(x, y), V. Usig the iverse Laplace trasformatio 3, we may write p(z) as ( ) ˆτ+j ˆω+j p(z) = dτ dω e κτω(x,y) τ T z, (9) πj ˆτ j ˆω j where κ τω (x, y) is the joit cumulat geeratig fuctio of Z(x, y), V, give by κ τω (x, y) = log E Q e τz(x,y)+ωv, ad where τ = (τ, ω) T. For our particular Z(x, y) ad V, κ τω (x, y) is give by ( ) τ (y X) κ τω (x, y) = log E Q e (X) ωc. () (y x) e assume that ˆτ = (ˆτ, ˆω) is withi the regio of covergece of the complex itegratio (9). Now, we perform a Taylor expasio of κ τω (x, y) aroud ˆτ, i.e., κ τω (x, y) κˆτ ˆω (x, y) + (τ ˆτ ) T κ ˆτ ˆω(x, y)+ + (τ ˆτ )T κ ˆτ ˆω(y)(τ ˆτ ), () where κ τω(x, y) ad κ τω(y) are the gradiet ad the Hessia matrix of κ τω (x, y), respectively give by κ τω(x, y) = κ τω(y) = τ ω τ τω ωτ τ κ τω (x, y), () κ τω (x, y). (3) e ote that κ τω(y) does ot deped o x, as the term (y x) i () is liear with τ. Pluggig () ito (9) ad makig the chage of variables ˆτ + jτ i = τ ad ˆω + jω i = ω, we obtai that the probability desity fuctio p(z) is approximated as p(z) e κˆτ ˆω(x,y) ˆτ T z ( ) dτ i dω i e jτ T i z ϕ(τ i ), (4) π where ϕ(τ i ) is the characteristic fuctio of a bidimesioal ormal distributio with mea κ ˆτ ˆω (x, y) ad covariace matrix κ ˆτ ˆω (y), i.e., ϕ(τ i ) = e jτ T i κ ˆτ ˆω (x,y) τ T i κ ˆτ ˆω (y)τ i. (5) Hece, sice ϕ(τ i ) is itegrable i R, solvig the itegratio (4) leads to the saddlepoit approximatio 4 of p(z), i.e., p(z) e κˆτ ˆω(x,y) ˆτ T z e (z κ ˆτ ˆω (x,y))t (κ ˆτ ˆω (y)) (z κ ˆτ ˆω (x,y)) (π) κ ˆτ ˆω (y), (6) Sice κ τω(x, y) ad κ τω(y), grow liearly with, for sufficietly large we may eglect the terms i z i the quadratic form i (6), i.e., p(z) eκˆτ ˆω(x,y) ˆτ T z κ ˆτ ˆω (x,y)t (κ ˆτ ˆω (y)) κ ˆτ ˆω (x,y) (π) κ ˆτ ˆω (y). (7) Usig the approximatio (7) ito (8), we obtai pep(x, y) dz δ δ e ˆτ T z dv eκˆτ ˆω(x,y) κ ˆτ ˆω (x,y)t (κ ˆτ ˆω (y)) κ ˆτ ˆω (x,y) (π) κ ˆτ ˆω (y). (8) Solvig the itegratio w.r.t. z, we obtai that for a give chael iput x ad chael output y, the pairwise error probability (4) uder the radom codig esemble () ca be approximated by pep(x, y) γ (x, y) e κτω(x,y), (9) where we redefie ω = ˆω ad τ = ˆτ for otatio clarity, κ τω (x; y) is the cumulat geeratig fuctio (), ad γ (y) is a subexpoetial related to () ad (3) as e κ τω (x,y)t κ τω (y) κ τω (x,y) e γ (x, y) = e ωδ ωδ. τω (π) κ τω(y) () e remark that () ivolves the expectatio accordig to the i.i.d. distributio Q (x) = i= Q(x). e also ote that the optimal auxiliary parameters τ ad ω would be chose as the uique miimizers of κ τω (x, y), which would set κ τω(x, y) = i the Taylor expasio () ad i (). However, this requires oe optimizatio for every x ad y. Istead, we let τ ad ω be fixed for every x ad y, at the cost of havig ozero κ τω(x, y) i γ (x, y). As reported i, 5 for the power-costraied AGN chael, decays subexpoetially i, hece ot affectig the expoet. The saddlepoit approximatio (9) states that the pairwise error probability, uder cost-costraied i.i.d. radom codig esemble (), decays expoetially as e κτω(x,y), with a pre-factor γ (x, y). e ote that κ τω (x, y) = i τω(x; y) is the egative tilted iformatio desity i τω(x; y) = log E Q B. Radom Codig Uio Boud (y x) τ. () (y X) τ e ωc (X) e start by usig the idetity Emi{, A} = PA U, where U is a uiformly distributed radom variable i the, iterval, to write the rcu(m) as rcu(m) = E P F {(X, Y, U) R } ()

3 where R is the set R = {(x, y, u) : log(m ) + log pep(x, y) log u}, (3) ad F (u) is the uiform probability distributio. Usig the right had side of (), we further have that rcu(m) = E Q F {(X, Y, U) R, X D }. (4) Similarly to Sec. III-A, we defie the radom variables Z = log(m ) + log pep(x, Y ) log U ad V = c (X), ad use the defiitios of R ad D to write equatio (4) for a strogly o-lattice two-dimesioal radom variable, i.e., rcu(m) = dz δ δ dv p(z), (5) where z = (z, v) T, ad p(z) is the joit probability desity fuctio of (Z, V ), give by ( ) ˆρ+j ˆλ+j p(z) = dρ dλ e χ(ρ,λ) ρt z. (6) πj Now, ρ = (ρ, λ) T, ad χ(ρ, λ) = log E Q F e ρz+λv is the joit cumulat geeratig fuctio of Z, V, i.e., χ(ρ, λ) = log E Q F (M ) ρ pep(x, Y ) ρ U ρ e λc (X). (7) Pluggig the saddlepoit approximatio (9) ito (7), takig log(m ) R, ad after some mathematical maipulatios, we obtai that χ(ρ, λ) ρr log( ρ)+ + log E Q (γ (X, Y )e κτω(x,y )) ρ e λc (X). (8) The saddlepoit approximatio of the pairwise error probability was foud by directly expadig the cumulat geeratig fuctio (), as both Z ad V were the sum of i.i.d. radom variables. This is ot the case of χ(ρ, λ), as there are terms i (8) that are ot liear with. It will prove coveiet to write equatio (8) as χ(ρ, λ) ρr + log E Q e ρκτω(x,y ) e λc (X) + + log E Q ρλ ρλ γ (X, Y ) ρ log( ρ), (9) where Q ρλ (x) ρλ (y x) is the tilted distributio Q ρλ(x)ρλ(y x) = Q (x) (y x)e ρκτω(x,y) e λc (x), ν (3) beig ν a ormalizatio factor. Sice log( ρ) ad γ (x, y) do ot grow expoetially with, we obtai that where lim χ(ρ, λ) = ρr E (ρ, λ), (3) E (ρ, λ) = lim log E Q e ρκτω(x,y ) e λc (X). (3) For a memoryless chael (y x) ad i.i.d. iput distributio Q (x), settig τ = +ρ ad ω = λ, turs equatio (3) ito Gallager s E fuctio for costraied iputs, Eq. (7.3.43) give by E (ρ, λ) = log ( +ρ Q(x) (y x) +ρ e λc(x)). (33) y x Defiig θ (ρ, λ) = E Q ρλ ρλ γ (X, Y ) ρ, (34) the cumulat geeratig fuctio (7) is of the form χ(ρ, λ) ρr E (ρ, λ)+log θ (ρ, λ) log( ρ). (35) Usig (35) ito (6), ad the solvig (5), we obtai that ( ) ˆρ+j ˆλ+j rcu(m) = dρ dλ πj e λδ e λδ λ eρr E(ρ,λ) θ (ρ, λ). (36) ρ( ρ) The saddlepoit approximatio to the rcu(m) the ivolves expadig ρr E (ρ, λ) aroud ˆρ = (ˆρ, ˆλ), the uique miimizers such that ρ E (ρ, λ) = R, (37) ρ=ˆρ λ E (ρ, λ) =. (38) λ=ˆλ Therefore, aroud ˆρ, we fid the expasio ρr E (ρ, λ) ˆρR E (ˆρ, ˆλ)+ (ρ ˆρ)T Vˆρˆλ(ρ ˆρ), (39) where Vˆρˆλ, the cost-costraied dispersio matrix, is the Hessia matrix of ρr E (ρ, λ), give by ρ Vˆρˆλ = ρλ E (ρ, λ). (4) λρ λ ρ=ˆρ,λ=ˆλ The covergece of equatio (36) highly depeds o the poles at ρ =, ρ = ad λ =. As discussed i, Theorem 7.3., the error expoet of the costraied iput radom codig esemble is give by E(R, Q) = if ρr E (ρ, λ). (4) ρ,λ he < ˆρ < ad ˆλ >, this correspods to rates R betwee the critical rate R (Q), defied as the rate for which E(R, Q) is achieved at ˆρ =, ad the mutual iformatio I(Q), give by I(Q) = E Q log (y x) (y X), (4) E Q for which ˆρ = ˆλ =. For this rage of ˆρ ad ˆλ, the complex itegratio (6) coverges for ay z, so that we ca use the Taylor expasio (39) to approximate (36). Coversely, if the rate R is such that the parameter ˆρ satisfyig (37) lies outside

4 the (, ) iterval, we eed to shift the itegratio axis of ˆρ at the cost of itroducig additioal terms due to the Cauchy s residue theorem 3. For sake of clarity, we cosider the more explaatory case of ρ (, ). To solve (36), it is coveiet to use the idetity ρ( ρ) = ρ + ρ, (43) ad ote that we ca equivaletly write (5) as rcu(m) = δ ( dv p (z)dz + δ ) p (z)dz, (44) where the probability distributios p (z) ad p (z) are respectively give by p (z) = p (z) = ( ) ˆρ+j πj dρ ( ) ˆρ+j dρ πj ˆλ+j ˆλ+j dλ e ζ(ρ,λ)+z ρt z, (45) dλ e ζ(ρ,λ) ρt z. (46) beig ζ(ρ, λ) ρr E (ρ, λ) + log θ (ρ, λ). Usig the Taylor expasio (39) ito (45) ad (46), ad followig the footsteps of the derivatio of equatios (4), (5) ad (6), we obtai the followig saddlepoit approximatios 4 p (z) e ζ(ˆρ,ˆλ)+z ˆρ T z (π) Vˆρˆλ e zt V ˆρˆλ z, (47) p (z) e ζ(ˆρ,ˆλ) ˆρ T z (π) Vˆρˆλ e zt V z ˆρˆλ. (48) Usig the approximatios (47) ad (48) ito (44) ad defiig the bidimesioal itegratio itervals I = (, ) (δ, δ ) ad I = (, ) (δ, δ ), we obtai that the saddlepoit approximatio of the radom codig uio boud to the error probability (3) uder the cost-costraied radom codig esemble () is give by rcu(m) α e (E(ˆρ,ˆλ) ˆρR), (49) where the factor α is foud as α = θ (ˆρ, ˆλ) ( dz e z ˆρ T z zt V z ˆρˆλ + I (π) Vˆρˆλ + dz e ˆρ T z zt V ˆρˆλ I (π) Vˆρˆλ z ), (5) ad (ˆρ, ˆλ) are the saddlepoits obtaied from (37) (38). I (5), z is a bidimesioal itegratio variable, V ρλ is the costcostraied dispersio matrix give by (4), ad the parameter θ (ˆρ, ˆλ) is computed as (34), where γ (x, y) is give by () with ω = ˆλ ad τ = +ˆρ ad the expectatio is uder the tilted distributio Q ρλ (x) ρλ (y x) give i (3). Computig α ivolves solvig two bidimesioal itegratios, which ca be doe usig stadard umerical itegratio packages. A simpler expressio is obtaied by further expadig α as. Neglectig the quadratic form zt V z ˆρˆλ i (5) to solve the bidimesioal itegratios, we obtai that α ca be asymptotically approximated by e ˆλδ e ˆλδ θ (ˆρ, α ˆλ) ˆλˆρ( ˆρ) (π) Vˆρˆλ. (5) Clearly, for rates R satisfyig R (Q) < R < I(Q), the saddlepoit approximatio (49) recovers the correct expoetial decay of the error probability, i.e. (4). Furthermore, sice κ τω(y) ad V ρλ both grow as, ad decays as 5, it follows that α i the expasio (5), ad hece also the cost-costraied radom codig error probability (49), + ˆρ have a polyomial decay of, exactly the same as the ucostraied case 9, Eq. (34). IV. AGN CHANNEL ITH SPHERICAL CODES e umerically evaluate the saddlepoit approximatio (49) for the shell iput AGN chael, where codewords satisfy i= x i = P, beig P the power, ad the AGN chael trasitio probability is give by (y x) = (y πσ e i x i ) σ, (5) i= beig σ the oise power. e defie the sigal-to-oise ratio (SNR) as sr = P σ. Shao gave a expressio for the exact RCU boud based o packig spheral coes, Eq. (9), whose accurate computatio is challegig, ad a asymptotic approximatio for large, give by, Eq. (5). I order to map our model ito the AGN chael with spherical codig, we cosider the fuctio c (x) = i= x i P, ad the set D = {x : P δ i= x i P }, correspodig to δ = δ ad δ = i (). e further cosider the distributio Q (x) = i= δ 4πP πp e x P. (53) Uder this settig,, Eq. (7.3.9). Particularizig our expressios, ad takig the limit as δ to costrai our code to be spherical, we obtai that the radom codig uio (RCU) boud ca be approximated as (49) where the Gallager fuctio E (ρ, λ) is E (ρ, λ) = λp ( + ρ) + log( λp )+ + ρ ( log λp + sr ), (54) + ρ ad α is give by α = θ (ˆρ, ˆλ) P V ( ( ) ( erfcx ( ˆρ) V Vˆρˆλ + erfcx ˆρ V )). Vˆρˆλ (55)

5 I (55), erfcx(t) is a modified Gaussia error fuctio give by erfcx(t) = sig(t) erfc( t ) t e, V is give by V = λ E (ρ, λ), (56) (ρ,λ)=(ˆρ,ˆλ) ad θ (ˆρ, ˆλ) ca be computed i closed-form from equatios (34), () ad (3), where κ τω (x, y) is give by κ τω (x, y) = ( ωp )τ ( ωp )σ + P τ y ωp + + τ σ y x log ( ωp + τ sr). (57) The expressio of θ (ˆρ, ˆλ), cumbersome ad ot particularly iformative, is ot icluded for the sake of space limitatios. Either expadig (55) as, or evaluatig (5) as δ, we obtai that the saddlepoit approximatio asymptotically behaves as (49), where the Gallager fuctio E (ρ, λ) is give by (54), ad ow α is give by α θ (ˆρ, ˆλ) ˆρ( ˆρ) πp Vˆρˆλ. (58) e have derived two approximatios of the RCU boud for the shell iput AGN chael, both of the form (49). e deote the approximatio by RCU (saddlepoit) for α give by (55), ad RCU (asymptotic) for α is give by (58). Figure shows our two approximatios, together with the exact Shao boud, Eq. (9), ad the asymptotic Shao boud, Eq. (5). e choose a rate R at 9% of I(Q) =.86 bits/chael use, satisfyig R (Q) < R < I(Q), where I(Q) = log ( + sr ), (59) R (Q) = ( sr log 4 + sr ). (6) For completeess, we also iclude the Shao lower boud, Eq. (5). As it ca be see from the figures, all curves are very close to each other. A more iformative parameter is the fourth-order term of the error probability expasio, deoted as β rcu(m) β +ˆρ e (E(ˆρ,ˆλ) ˆρR), (6) ad show i Figure. The umerical results suggest that the saddlepoit approximatio (49) with α i (55) is a alterative expressio for the asymptotic Shao boud, Eq. (5). The mai advatage of our saddlepoit approximatio is that it is applicable to cost-costraied memoryless chaels uder some strogly o-lattice coditios. REFERENCES C. Shao, A mathematical theory of commuicatio, Bell Syst. Tech. Joural, vol. 7, pp , May 948. R. Gallager, Iformatio Theory ad Reliable Commuicatio. Joh iley & Sos, A. Gati, A. Lapidoth, ad I. E. Telatar, Mismatched decodig revisited: Geeral alphabets, chaels with memory, ad the wide-bad limit, IEEE Tras. If. Theory, vol. 46, o. 7, pp , Nov.. 3 I. Csiszár ad J. Körer, Iformatio Theory: Codig Theorems for Discrete Memoryless Systems, d ed. Cambridge Uiversity Press,. Error probability, ɛ(m) , Code legth, RCU (asymptotic) RCU (saddlepoit) Shao (asymptotic) Shao (exact) Shao (lower boud) Fig.. Radom codig error probability bouds versus the code legth at a rate R =.958 bits/chael use, ad sr = 5 db. Fourth order term, β , Code legth, RCU (asymptotic) RCU (saddlepoit) Shao (asymptotic) Shao (exact) Shao (lower boud) Fig.. Fourth order term β versus the code legth at a rate R =.958 bits/chael use, ad sr = 5 db. 5 S. Shamai ad I. Saso, Variatios o the Gallager bouds, coectios, ad applicatios, IEEE Tras. If. Theory, vol. 48, o., pp , Dec.. 6 R. Fao, Trasmissio of Iformatio: A Statistical Theory of Commuicatios. M.I.T. Press, J. L. Jese, Saddlepoit Approximatios. Oxford Uiversity Press, Y. Altuğ ad A. B. ager, Refiemet of the radom codig boud, IEEE Tras. If. Theory, vol. 6, o., pp , 4. 9 J. Scarlett, A. Martiez, ad A. Guillé i Fàbregas, Mismatched decodig: Error expoets, secod-order rates ad saddlepoit approximatios, IEEE Tras. If. Theory, vol. 6, o. 5, pp , 4. T. Erseghe, Codig i the fiite-blocklegth regime: Bouds based o Laplace itegrals ad their asymptotic approximatios, IEEE Tras. If. Theory, vol. 6, o., pp , Dec. 6. C. Shao, Probability of error for optimal codes i a Gaussia chael, Bell Syst. Tech. Joural, vol. 38, o. 3, pp , May 959. Y. Polyaskiy, H. V. Poor, ad S. Verdú, Chael codig rate i the fiite blocklegth regime, IEEE Tras. If. Theory, vol. 56, o. 5, pp , May. 3 G. Doetsch, Itroductio to the Theory ad Applicatios of the Laplace Trasformatio. Spriger, Feller, A Itroductio to Probability Theory ad its Applicatios. New York, NY: iley, 97, vol..

Maximum Empirical Likelihood Estimation (MELE)

Maximum Empirical Likelihood Estimatio (MELE Natha Smooha Abstract Estimatio of Stadard Liear Model - Maximum Empirical Likelihood Estimator: Combiatio of the idea of imum likelihood method of momets,