Statistical Assessment of Model Fit for Synthetic Aperture Radar Data

Transcription

1 Statistical Assessment of Model Fit for Synthetic Aperture Radar Data Michael D. DeVore a and Joseph A. O Sullivan b Electronic Systems and Signals Research Laboratory Department ofelectrical Engineering, Washington University, St. Louis, MO 633 ABSTRACT Parametric approaches to problems of inference from observed data often rely on assumed probabilistic models for the data which may be based on knowledge of the physics of the data acquisition. Given a rich enough collection of sample data, the validity of those assumed models can be assessed in a statistical hypothesis testing framework using any of a number of goodness-of-fit tests developed over the last hundred years for this purpose. Such assessments can be used both to compare alternate models for observed data and to help determine the conditions under which a given model breaks down. We apply three such methods, the χ test of Karl Pearson, Kolmogorov s goodness-of-fit test, and the D Agostino-Pearson test for normality, to quantify how well the data fit various models for synthetic aperture radar (SAR) images. The results of these tests are used to compare a conditionally Gaussian model for complex-valued SAR pixel values, a conditionally log-normal model for SAR pixel magnitudes, and a conditionally normal model for SAR pixel quarter-power values. Sample data for these tests are drawn from the publicly released MSTAR dataset. Keywords: goodness-of-fit, synthetic aperture radar, model assessment, conditionally Gaussian. INTRODUCTION Many approaches to problems of detection, classification, and estimation from measured sensor data are based upon statistical models for the observations. These models may be developed through an understanding of the phenomenological properties of the scene, behavior of the sensing device, and simplifying assumptions and approximations. Performance of the inference algorithms derived from the assumed models depends heavily on the extent to which the resulting models accurately describe the actual sensor measurements. Algorithms developed from inaccurate models may yield suboptimal performance. Further, methods for predicting the performance of inference algorithms often rely on these same models and the quality of such predictions can be expected to suffer if the model is a poor match to actual data. It is therefore desirable to quantify, from a collection of sample observations, how accurately the model and data fit. We are particularly concerned with statistical models for synthetic aperture radar (SAR) imagery that can be used to support inference algorithms. A number of methods for quantitatively assessing the validity of statistical models in light of sample data have been developed over the last hundred years. Many of these methods place the problem in a statistical hypothesis testing framework, pitting a null-hypothesis H, an assertion that the data were generated according to the model, against an alternative hypothesis H A, an assertion that they were not. The methods are then implemented by computing some statistic of the random observations that has a known distribution if H were true. The quantitative result of the method is the probability that the chosen statistic would take a value at least as extreme as the value computed if H were true. Values of this quantity, often called the P-value, close to zero are interpreted as evidence that H should be rejected in favor of H A. These methods are not to be confused with methods for model selection which seek the model that best describes observed data from a set of specified models, irrespective of whether any model is actually a good fit to the data. All of the models we consider involve the Gaussian distribution. There are mixed views represented in the literature as to which tests for the Gaussian distribution are the most powerful, that is are the most capable of properly rejecting H for sample data drawn from non-gaussian distributions. We therefore employ three tests for goodness-of-fit, each of which relies on differing properties of Gaussian random variables. K. Pearson s χ test, a mdd@cis.wustl.edu, b jao@ee.wustl.edu Algorithms for Synthetic Aperture Radar Imagery VIII, Edmund G. Zelnio, Editor, Proceedings of SPIE Vol. 438 () SPIE X//$5. 379

2 appropriate for categorical data or binned continuous-valued data, is an approximate test based on the number of random observations in each category. It effectively tests whether the histogram of sample data is reasonable under the assumed distribution. The Kolmogorov-Smirnov test, appropriate for continuous-valued ordinal data, is based on the largest magnitude difference between the empirical cumulative distribution function (CDF) and the CDF of the assumed distribution. The test of R. D Agostino and E. Pearson is a test specifically for departures from normality based on the distribution of sample skewness and kurtosis values. We consider the assessment of three distinct statistical models for SAR imagery. SAR data consist of an array of pixels with complex values that are related to the electro-magnetic reflectivity of regions in the scene being imaged. DeVore and O Sullivan 4 compare a conditionally-gaussian model for complex-valued SAR pixels with log-normal and quarter-power normal models for pixel magnitudes in terms of the empirical performance delivered by classification algorithms derived from each. We assess the validity of generalized variants of these models relative to actual SAR data made publicly available through the Moving and Stationary Target Acquisition and Recognition (MSTAR) program. The analysis is performed for pixels corresponding to man-made vehicles as well as for pixels from natural clutter regions. Many statistical models for the marginal distribution of SAR image pixels have recently been addressed in the literature. Schnidman 5 considers a model for pixel squared-magnitudes with a three parameter mixture of noncentral χ distributions with noncentrality parameter governed by a gamma distribution and graphically demonstrates the fit to clutter imagery. Billingsley, et al. assess the fit of Rayleigh, Weibull, log-normal, and K-distributions to pixel magnitudes in clutter data and show via the Kolmogorov-Smirnov test that none fit well over the entire range of magnitudes. Kuruoglu and Zerubia 9 graphically demonstrate the fit of α-stable distributions to pixel magnitudes. These differ from the current study in their focus on marginal distributions across all pixels in an image and across all contents of the scene. Each of the three models we consider is a statistical characterization of individual pixels in a SAR image conditioned on the contents of the scene and the angle from which they are imaged. This is a distinction that Nathanson refers to as temporal distribution as opposed to a spatial distribution. A large number of images is available, however the number of images of each target from any given orientation is small. Thus, hypothesis testing must be carried out over a large number of small samples and the results aggregated for analysis. The P-value for each small-sample test is a uniformly distributed random variable on [, ] if H were true. Section describes the three conditional models for SAR imagery to be analyzed. Section 3 presents a description of the analysis methods to be employed. A description of the SAR data used in the analysis is provided in Section 4 along with details about aggregating the P-values from large numbers of test. Results of the statistical analysis are presented in Section 5 and conclusions follow in Section 6.. CONDITIONAL MODELS FOR SAR IMAGERY The conditionally complex-gaussian model for SAR imagery is based on the assumption that the region in the scene corresponding to each pixel i contains multiple scattering centers. The complex reflections from each combine at the SAR processor where they are further combined with additive white Gaussian noise, resulting in real and imaginary components that are independent Gaussian random variables. It is assumed that no single scattering center dominates the return so the pixels are modeled as zero-mean. The variance is common to both real and imaginary components but can vary with pixel location, orientation of the radar platform with respect to the scene, and contents of the scene. Following the assumption that distinct pixels correspond to nonoverlapping regions of the scene, the pixels are modeled as independent from one another. The probability density function for each pixel value R i is then /σ p(r i )= πσi (a, i (a,θ), () θ)e ri where i denotes the pixel location, a describes the contents of the scene, and θ characterizes the position and orientation of the radar platform relative to the scene. The assumption of pixels with zero mean can be relaxed by introducing a mean value µ i (a, θ) that varies by pixel, scene contents, and sensor pose. In this case, the probability density for R i canbewrittenas p Ri (r i )= /σ i (a,θ). () πσi (a, θ)e ri µi(a,θ) The conditionally log-normal model has been suggested as a good empirical fit to radar returns from clutter and may be reasonable for modeling multiple scattering effects. 4 The log-normal distribution has long been recognized 38 Proc. SPIE Vol. 438

3 as a good fit to observations consisting of a product of a large number of independent observations. 7 Under this model, the magnitude of each pixel value follows a log-normal distribution so that the logarithm of the magnitude is Gaussian with mean and variance that depend on pixel location, scene contents, and sensor pose. Following the nonoverlapping region assumption, the pixels are taken to be independent from one another. The distribution of the logarithm of pixel magnitude, X i =log R i is p Xi (x i )= πσ i (a, θ) e (xi µi(a,θ)) /σ i (a,θ), (3) where µ i (a, θ) andσi (a, θ) denote the mean and variance, respectively, of pixel i in an image of a scene with contents a taken from pose θ. The conditionally quarter-power normal model stems from the empirical observation 6 that some power transformations, with parameters between and, of gamma distributed random variables share some statistical properties, notably kurtosis, with Gaussian random variables. The quarter-power normal model assumes that the magnitude of pixel values follows a gamma distribution conditioned on pixel location, scene contents, and pose of the radar platform. The square-root of this magnitude, X i = R i is modeled as following a Gaussian distribution. The density function for X i has the form of (3) where mean and variance parameters now characterize the square-root of pixel magnitude rather than its logarithm. 3. GOODNESS-OF-FIT TESTS Given a set of observations R,R,...,R N with identical distributions, statistical goodness-of-fit testing procedures can be used to choose between hypothesis H, an assertion that the data were drawn from distribution q, andan alternate hypothesis H A, an assertion that the data were not drawn from q. A test statistic T (R,R,...,R n )that is a function of the observations is determined such that the distribution of T is known under the assumption that H were true. The probability that T takes a value at least as extreme as its observed value t under this distribution is reported as the P-value of the hypothesis test. Because it is a probability, the P-value can take on values between zero and one, inclusive. Extremely small P-values are taken as evidence that the data do not represent a sample drawn from q while large P-values indicate a lack of evidence to support rejecting H. In the case that the data are drawn from q, the P-value will be uniformly distributed. Powerful tests, tests highly effective at correctly rejecting H, are characterized by a distribution of P-values that is sharply skewed toward zero when the data are not drawn from q. K.Pearson sχ test of fit is an approximate test for the multinomial distribution of categorical data. Application to the continuous-valued observations collected by SAR platforms requires that the range of observations be discretized and the observations binned accordingly. We can partition the set of real numbers into K intervals such that, if actually drawn from q, the expected number of the N observations to fall in interval k is greater than one. Conservative recommendations state that the expected number of observations within each category should be at least five. If we let q k denote the integral of q over the kth such interval, then the expected number of observations in the interval is Nq k.letn k denote the observed number of observations in interval k, then the statistic Dχ given by Dχ = K (N k Nq k ) (4) Nq k k= is asymptotically χ distributed with K degrees of freedom for N if H is true. Large values of D indicate that the histogram of the observed data does not match the expected histogram under q and suggest rejecting H. The probability that Dχ will exceed the observed value d is approximately equal to the right-tailed probability Pr [ Dχ >d] P χ (d ). (5) K This probability is reported as the P-value of the test. The Kolmogorov-Smirnov test is based upon the empirical cumulative distribution function of the observed data. The empirical CDF is a piecewise constant function ˆP R (r) thatequalszeroatr = and increases by a value of /N at each observation R k.thatis, ˆP R (r) = N {R k : R k r}, (6) Proc. SPIE Vol

4 where S denotes the cardinality of the set S. The Kolmogorov statistic, D KS, is defined as the supremum of the magnitude difference between the empirical CDF and the cumulative distribution under H, Q R (r). That is, D KS =sup ˆP R (r) Q R (r). (7) r An exact expression for the distribution of D KS under H was derived by Kolmogorov and can be readily approximated numerically as explained in Press, et al. 3 Large observed values d of the statistic D KS suggest evidence that H should be rejected and the probability Pr [D KS >d] is reported as the P-value of the test. R. D Agostino and E. Pearson suggest a test for departure from normality based upon estimates of the skewness, γ,andkurtosis,γ, from the sample data. R. Fisher defined skewness and kurtosis in terms of cumulants of a distribution as γ = κ 3 and γ κ 3/ = κ 4 κ. (8) Sample skewness and kurtosis values are computed in terms of the corresponding k-statistics, which are defined as symmetric unbiased estimators of distribution cumulants. 8 That is, where g = k 3 k 3/ and g = k 4 k, (9) k = Ri () N k = (Ri k ) () N N k 3 = (Ri k ) 3 () (N )(N ) k 4 = N(N +) (R i k ) 4 3(N )( (R i k ) ). (3) (N )(N )(N 3) Closed form expressions for the distribution of g and g are not readily available for Gaussian distributed observations. Fisher 5 notes The exact distribution for N>3seems not to be expressible simply in terms of known functions. A number of attempts have been made at approximating the distributions based upon moments of the distributions of g and g for varying N and these are generally evaluated empirically. For this study, we have generated empirical cumulative distributions P G and P G from one million samples of standard normal random variables for many values of N. The test statistic for the D Agostino-Pearson test is defined in terms of these functions as D DP = [ Φ (P G (g )) ] [ + Φ (P G (g )) ], (4) where Φ (z) is the inverse cumulative distribution function for a standard normal random variable. The statistic D DP is not χ distributed because the two terms in the expression are not independent. We have also generated empirical cumulative distributions of observations d of the statistic D DP. Large values of the test statistic suggest evidence that H should be rejected and the probability Pr [D DP >d]isreportedasthep-valueofthetest. 4. LARGE NUMBERS OF SMALL SAMPLES We apply the goodness-of-fit tests described in the previous section to publicly available SAR data collected as part of the MSTAR program. Table summarizes the data used in the analysis and Figure contains three representative images from the set on a log-magnitude scale. The data represent images of ten different vehicle classes imaged from two depression angles and are fairly evenly distributed over 36 of azimuth. The SAR images have a vehicle located near the center which is surrounded by background clutter. Model fitness assessments are performed separately on vehicle and clutter pixels by restricting the tests to pixel regions centered in the images and in the upper left-hand corner, respectively. The models described in Section are defined in terms of mean and variance functions of scene content and relative pose. For our purposes, the scene content a will refer to the class of target in the image and pose θ will refer 38 Proc. SPIE Vol. 438

5 Target Vehicles Images (5 Depression Images (7 Depression) ) S b BMP- 9563, 9566, c BRDM- E BTR-6 kyt BTR-7 c D7 9v T6 A T-7 3, 8, s ZIL3 E ZSU 3 4 d Table. MSTAR dataset used in statistical analysis. In total, 686 images of ten targets at two depression angles are used. Figure. Magnitude images of vehicles in the MSTAR dataset shown on logarithmic scale. From left to right the images are of the S, D7, and ZSU 3 4 imaged at 5 depression and with 45 azimuth relative to the radar. Proc. SPIE Vol

6 to the azimuth angle of the target relative to the radar platform. We assume that all images have been registered in a consistent manner so that shifts of the vehicle within the slant-plane can be disregarded. Target azimuth is a continuous valued parameter and we do not expect to have multiple images of each vehicle taken from the same azimuth angle. We therefore approximate the mean and variance functions as relatively stable over small changes in azimuth and consider the values at a fixed pixel location in images taken within of each other to be samples from the same distribution. For each vehicle class, we collect samples from 6 such intervals of evenly distributed over the 36 of possible azimuth. The tests of Section 3 are applied to the observations for each combination of pixel location, azimuth interval, and vehicle class. For tests of the conditionally Gaussian model, which explicitly models both the real and imaginary components of pixel values, each component is tested separately and the resulting P-values are combined so that the tests of each model will have approximately the same powers. If the model assumptions and approximations are valid, each such test should result in a P-value that is uniformly distributed on [, ] and the empirical cumulative distribution of P-values should approximate a straight line with slope on that interval. The D Agostino-Pearson test computes all necessary values directly from the sample data. The conditionally Gaussian model involves two variants, one with a mean that is assumed to be identically zero. To directly support tests of both variants, empirical distributions of g, g,andd DP were generated for both cases, substituting κ = in place of the first k-statistic in Equations () through (3) in the zero-mean case and modifying k through k 4 to retain unbiased estimates. The χ test discussed in Section 3 can be modified to accommodate unknown distribution parameters by substituting estimated values into the distribution q and decreasing the degrees of freedom of Dχ by one for each estimated value. However, many of the unique combinations of pixel location, azimuth interval, and vehicle class will have only a small number of samples. The requirement that each category of binned data should contain at least five samples will imply that only two bins can be employed which will not yield a powerful test for normality. Further, the Kolmogorov-Smirnov test does not accommodate estimated distribution parameters. A solution to these problems is to convert each sample into a distribution that is known exactly under the assumption that H is true and applying the χ and Kolmogorov-Smirnov tests to these transformed samples. If Z,Z,...,Z N are Gaussian distributed with mean µ and unknown variance, then the random variables (Z k µ)/ξ(k) follow the Student s T distribution with N degrees of freedom, where ξ(k) is an estimate of variance neglecting sample Z k as in ξ (k) = N (Z j µ). (5) Similarly, if the mean is unknown, the random variables (Z k Z)/ N S (k) follow the Student s T distribution with N degrees of freedom, where Z is the sample mean of the Z k,ands(k) is an estimate of variance neglecting sample Z k as in S(k) = (Z j Z (k) ) and Z (k) = Z j. (6) n n j k j k N The distribution of the transformed data depends only the size of the sample but results in data that are not necessarily independent. For any combination of azimuth interval and vehicle class all such transformations for each of the pixel locations will result in samples from the same distribution. Since the models hold that the values in pixel locations are independent from one another, randomly chosen collections of the transformed samples from the pixel regions should appear as a sample of independent observations. Further, we can choose collections large enough to satisfy the bin requirements of the χ test. The χ and Kolmogorov-Smirnov tests become tests of whether the transformed data follows the T distribution with appropriate degrees of freedom. 5. RESULTS Figure shows the cumulative distribution of P-values from pixels in the vehicle region of the images for each of the models described in Section when assessed by each of the tests in Section 3. Models which highly accurately describe the data should yield cumulative distributions that approximate a diagonal line through the plot center for each of the three tests. For any P-value, α, the corresponding cumulative probability indicates the fraction of samples that would fail a test with significance α. j k 384 Proc. SPIE Vol. 438

7 D Agostino-Pearson.. Kolmogorov-Smirnov.... Pearson s χ Figure. Empirical cumulative distribution of the P-values for model fitness tests of pixels in the central (vehicle)region of SAR images. Each panel shows the P-value cumulative distribution for complex-gaussian (with and without zero-mean assumption), log-normal, and quarter-power normal models. The left panel shows the distribution with the D Agostino-Pearson test, the right panel shows results from the Kolmogorov-Smirnov test, and the bottom panel shows results from Pearson s χ test. Proc. SPIE Vol

8 The test of D Agostino and Pearson indicates that, for most combinations of pixel locations, vehicle, and pose, sample skewness and kurtosis values are within the range reasonably expected under each of the assumed models. There is a slight skewness toward low probability combinations suggesting that the models are not an exact fit for the data. The Kolmogorov-Smirnov test and Pearson s χ test of the T transformed sample data similarly show that many combinations of vehicle and pose yield samples that are within the range reasonably expected. The distribution of P-values for these tests is much more skewed toward low probability events, however. This suggests that some model assumptions and/or approximations are violated more often than would be otherwise expected. The distributions of P-values for the zero-mean variant of the complex-gaussian model are more skewed toward low values than for the nonzero-mean case. Because tests enforcing the zero-mean constraint are more powerful than those without, this difference in distribution cannot, by itself, be taken as evidence that the zero-mean assumption is improper. For a more detailed analysis of the zero-mean assumption as well as other key model assumptions, see DeVore. 3 Figure 3 shows the distribution of P-values for the clutter region in the upper left corner of the images. Both complex-gaussian models and the quarter-power normal model exhibit distributions that are less skewed than for the pixels in the vehicle region. Since the tests were performed at equal powers, this suggests that the models may be a better fit for such clutter regions than for data drawn from the man-made vehicles. The log-normal model yields P-values that are skewed much more heavily toward low probabilities for the clutter data than for the vehicle data. The deviation from the expected uniform distribution is especially severe in the Kolmogorov-Smirnov and χ tests. This suggests that the conditionally log-normal model is not a good fit for the clutter data represented in the sample data. 6. CONCLUSIONS We have presented a method for assessing the degree to which conditional models of observed data fit large datasets with small samples in the conditioning variables. We have applied this method to four models for SAR data using publicly available imagery. With the exception of the log-normal model applied to clutter data, the majority of samples would pass a 5% significance test for all models considered. For clutter samples, nearly 9% of the samples would pass tests for the complex-gaussian and quarter-power normal models at the 5% significance levels. The tests revealed that none of the models are exact matches to the available data. This is to be expected because of the approximations to mean and variance functions to accommodate continuous target pose described in Section 4. The experiments suggest that the conditional log-normal model does not accurately characterize clutter pixels in the given dataset. This is consistent with the observation made by DeVore and O Sullivan 4 that recognition performance in algorithms derived from the log-normal distribution suffers if images to be classified contain an excessive amount of background clutter. In every test conducted the quarter-power normal model yielded a more even distribution of P-values, followed by the Gaussian and log-normal models, in turn. Since the tests were applied identically to all models, except for the more restrictive zero-mean Gaussian model, this would suggest that the quarter-power model may be a better fit to the SAR data considered. That the quarter-power normal model yielded fewer low probability events than did the complex-gaussian would seem to contradict the experiments in DeVore and O Sullivan 4 showing worse performance for recognition algorithms derived from the quarter-power model. However, the quarter-power algorithm investigated in that paper is based on an approximation that the variance of quarterpower values is constant across all pixel locations, vehicle types, and orientations. ACKNOWLEDGMENTS This work was supported in part by the US Army Research Office grant DAAH , the Office of Naval Research grant N , and the Boeing Foundation. REFERENCES. J. B. Billingsley, A. Farina, F. Gini, M. V. Greco, and L. Verrazzani. Statistical analyses of measured radar ground clutter data. IEEE Transactions on Aerospace and Electronic Systems, 35(): , Apr R. B. D Agostino and E. S. Pearson. Tests for departure from normality. Empirical results for the distributions of b and b. Biometrika, 6(3):63 6, Dec Proc. SPIE Vol. 438

9 D Agostino-Pearson.. Kolmogorov-Smirnov.... Pearson s χ Figure 3. Empirical cumulative distribution of the P-values for model fitness tests of pixels in the upper-left (clutter)region of SAR images. Each panel shows the P-value cumulative distribution for complex-gaussian (with and without zero-mean assumption), log-normal, and quarter-power normal models. The left panel shows the distribution with the D Agostino-Pearson test, the right panel shows results from the Kolmogorov-Smirnov test, and the bottom panel shows results from Pearson s χ test. Proc. SPIE Vol

10 3. M. D. DeVore. Recognition Performance from Synthetic Aperture Radar Imagery Subject to System Resource Constraints. PhD thesis, Washington University, May. 4. M. D. DeVore and J. A. O Sullivan. A performance-complexity study of several approaches to automatic target recognition from synthetic aperture radar images. IEEE Transactions on Aerospace and Electronic Systems. Submitted for publication. 5. R. A. Fisher. The moments of the distribution for normal samples of measures of departure from normality. Proceedings of the Royal Society of London, Series A, 3(A 8):6 8, K. Fukunaga. Introduction to Statistical Pattern Recognition, pages Academic Press, second edition, N. L. Johnson, S. Kotz, and N. Balakrishnan. Continuous Univariate Distributions, volume. John Wiley & Sons, M. Kendall and A. Stuart. The Advanced Theory of Statistics, volume. Macmillan Publishing Co., fourth edition, E. E. Kuruoglu and J. Zerubia. Modelling SAR images with a generalisation of the Rayleigh distribution. In M. B. Matthews, editor, Proceedings of the Thirty-Fourth Annual Asilomar Conference on Signals, Systems, and Computers, volume, pages 4 8, Oct... E. B. Manoukian. Mathematical Nonparametric Statistics. Gordon and Breach Science Publishers, F. E. Nathanson, J. P. Reilly, and M. N. Cohen. Radar Design Principles. McGraw-Hill, second edition, 99.. J. A. O Sullivan, M. D. DeVore, V. Kedia, and M. I. Miller. Automatic target recognition performance for SAR imagery using a conditionally Gaussian model. IEEE Transactions on Aerospace and Electronic Systems. Accepted for Publication. 3. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, second edition, H. R. Raemer. Radar System Principles. CRC Press, D. A. Shnidman. Generalized radar clutter model. IEEE Transactions on Aerospace and Electronic Systems, 35(3): , July Proc. SPIE Vol. 438