The following will be discussed in this chapter: LMMSE estimator: first step (obtaining unbiasedness), LMMSE estimator: second step (solve reduced problem), LMMSE estimator as projection, putting it all together, orthogonality relations, extension to multivariate case,. | Lecture Signals, systems & inference – Lecture 14: LMMSE estimation, orthogonality LMMSE estimation, orthogonality , Spring 2018 Lec 14 1 LMMSE estimator: first step (obtaining unbiasedness) Linear estimator: Yb` = aX + b , with a and b picked to minimize E[(Y - Yb` )2 ] over joint density of X and Y ) min E[(Y aX b)2 ] a,b | {z } Z First min E[(Z - b)2 ] ) b = µZ = µY - aµX b This yields an unbiased estimator: 2 E[ Yb` ] = E[Y ] = µY LMMSE estimator: second step (solve reduced problem) Now min E[(Y aX b)2 ] = E[({Y µY } a{X µX })2 ] a | {z } | {z } e Y e X . min E[(Ye e )2 ] aX a YX Y ) a= 2 = ⇢Y X X X (can be shown in di↵erent ways, ., by vector picture) 3 LMMSE estimator as projection ' Y ' ' ˆ/ Y - aX = Y - Y cos-1 (r ) YX ' ' X aX For the optimum a, (Ye aXe) ? Xe ., E[(Ye e )X aX e] = 0 ) a=4 2 = ⇢Y X X X Putting it all together Y Yb` = yb` (X) = µY + ⇢ (X - µX ) X Yb` - µY X µX or equivalently =⇢ Y X 2 2 Also, the resulting MMSE is Y (1 ⇢ ) 5 Orthogonality relations Unbiasedness condition can be written as Y - Yb` ? 1 (or ? to any constant) We also know (Ye e) ? X aX e or equivalently Y Yb` ? X e or equivalently Y Yb` ? X Conversely, first + last above yield equations for a, b 6 Extension to multivariate case min E[(Y {a0 + ⌃L a X j=1 j j }) 2 ] a0 ,.,aL | {z } b` Y L First min ) a 0 = µY ⌃j=1 aj µXj a0 This ensures unbiasedness of the estimator. Now min E[(Ye ⌃L e j=1 j j ] a X ) 2 a1 ,.,aL 7 Applying orthogonality gives the “normal equations” h⇣ ⌘ i E Ye - ⌃L e e j=1 aj Xj Xi = 0 2 32 3 2 3 C X1 X1 CX1 X2 ··· CX 1 X L a1 C X1 Y 6 C X2 X1 CX2 X2 ··· CX 2 X L 76 a2 7 6 CX2 Y 7 6 76 7 6 7 6 76 7=6 7 4 . . . . 54 . 5 4 . 5 C XL X1 CXL X 2 ··· CX L X L aL CXL Y (CXX ) a = cXY MMSE: CY2 - cY X (CXX )-1 cXY = CY2 - cY X .a 8 MIT .
