Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization

Transcription

1 IS th Symp Sensing via Image Information, June 2011, Yokohama, Japan Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization Ryohei Nakamura, Teppei Marumo and Kenihi Kanatani Department of Computer Siene, Okayama University, Japan Abstrat We present a new tehnique for alibrating an ultra-wide fisheye lens amera by imposing the onstraint that ollinear points be retified to be ollinear, parallel lines to be parallel, and orthogonal lines to be orthogonal Exploiting the fat that line fitting redues to an eigenvalue problem, we do a rigorous perturbation analysis to obtain a Levenberg-Marquardt proedure for the optimization Doing experiments, we point out that spurious solutions exist if ollinearity and parallelism alone are imposed Our tehnique has many desirable properties For example, no metri information is required about the referene pattern or the amera position, and separate stripe patterns an be displayed on a sreen to generate a virtual grid, eliating the grid point detetion 1 Introdution Fisheye lens ameras are widely used for surveillane purposes beause of their wide angles of view They are also mounted on vehiles for various purposes inluding obstale detetion, self-loalization, and bird s eye view generation [9, 12] However, fisheye lens images have a large distortion, so that in order to apply the omputer vision tehniques aumulated for the past deades, one first needs to retify the image into a perspetive view Already, there is a lot of literature for this [2, 4, 6, 7, 10, 11, 12, 13, 16, 17] The standard tehnique is to plae a referene grid plane and math the grid image with the referene grid, whose preise geometry is assumed to be known [2, 3, 4, 6, 17] However, this approah is not very pratial for reently popularized ultra-wide fisheye lenses, beause they over more than 180 angles of view and hene any (even infinite) referene plane annot over the entire field of view This diffiulty an be irumvented by using the ollinearity onstraint pointed out repeatedly, first by Onodera and Kanatani [14] in 1992, later by Swaathan and Nayar [16] in 2000, and by Devernay and Faugeras [1] in 2001 They pointed out that amera alibration an be done by imposing the onstraint that straight lines should be retified to be straight This onstraint was used to alibrate fisheye lenses by Nakano, et al [11], Kase et al [7], and Okutsu et al [12] Komagata et al [8] further introdued the parallelism onstraint and the orthogonality onstraint, requiring that parallel lines be retified to be parallel and orthogonal lines to be orthogonal However, the ost funtion has been diretly imized by brute fore means suh as the Brent method and the Powell method [15] In this paper, we adopt the ollinearity-parallelismorthogonality onstraint of Komagata et al [8] and optimize it by eigenvalue imization The fat that imposing ollinearity implies eigenvalue imization and that the optimization an be done by invoking the perturbation theorem was pointed out by Onodera and Kanatani [14] Using this priniple, they retified perspetive images by gradient deent Here, we onsider ultra-wide fisheye lenses and do a more detailed perturbation analysis to derive the Levenberg- Marquardt (LM) proedure, urrently regarded as the standard tool for effiient and aurate optimization We also show by experiments that the orthogonality onstraint plays an essential role for ultra-wide fisheye lenses and that orret alibration annot be done only by imposing ollinearity or parallelism or both, pointing out the existene of a spurious solution This fat has not been known in the past ollinearity-based work [1, 7, 11, 12, 14, 16] For data aquisition, we take images of stripe patterns of different orientations on a large-sreen display by plaing the amera in various positions Like the past ollinearity-based methods [1, 7, 11, 12, 14, 16], our method is non-metri in the sense that no metri information is required about the amera position or the referene pattern Yet, many researhers pointed out the neessity of some auxiliary information For example, Nakano et al [10, 11] proposed vanishing point estimation using oni fitting to straight line images (reently, Hughes et al [4] proposed this same tehnique again) Okutsu et al [13] piked out the images of antipodal points by hand Suh auxiliary information may be useful to suppress spurious solutions However, we show that aurate alibration is possible without any auxiliary information if we do eigenvalue imization for the ollinearityparallelism-orthogonality onstraint Setion 2 desribes our imaging geometry model Setion 3 gives derivative expressions of the fundamental quantities, followed by a detailed perturbation analysis of the ollinearity onstraint in Se 4, of the parallelism onstraint in Se 5, and of the orthogonality onstraint in Se 6 Setion 7 desribes our Levenberg-Marquardt (LM) optimization proedure In Se 8, we show our experiments and an examples of real sene appliations In Se 9, we onlude 2 Geometry of ultra-wide fisheye lens We onsider reently popularized ultra-wide fisheye lenses with the imaging geometry modeled by the IS3-04-1

2 17th Symp Sensing via Image Information, June 2011, Yokohama, Japan r Figure 1 The imaging geometry of a fisheye lens and the inident ray vetor m stereographi projetion θ f m r = 2f tan θ 2, (1) where θ is the inidene angle (the angle of the inident ray of light from the optial axis) and r (in pixels) is the distane of the orresponding image point from the prinipal point (Fig 1) The onstant f is alled the foal length We onsider Eq (1) merely beause our amera is as suh, but the following alibration proedure is idential whatever model is used For a manufatured lens, the value of f is unknown or may not be exat if provided by the manufaturer Also, the prinipal point may not be at the enter of the image frame Moreover, Eq (1) is an idealization, and a real lens may not exatly satisfy it So, we generalize Eq (1) into the form r + a 1 ) 3 + a2 ) 5 + = 2f tan θ 2, (2) and detere the values of f, a 1, a 2, along with the prinipal point position Here, is a sale onstant to keep the powers r k within a reasonable numerial range (in our experiment, we used the value = 150 pixels) Sine a suffiient number of orretion terms ould approximate any funtion, the righthand side of Eq (2) ould be any funtion of θ, eg, the perspetive projetion model (f/ ) tan θ or the equidistane projetion model (f/ )θ We adopt the stereographi projetion model of Eq (1) merely for the sake of simple initialization for our amera In Eq (2), even power terms do not exist, beause the lens has irular symmetry and hene r is an odd funtion of θ We assume that the azimuthal angle of the projetion is equal to that of the inident ray In the past, these two were often assumed to be slightly different, and geometri orretion of the resulting tangential distortion was studied Currently, the lens manufaturing tehnology is highly advaned so that the tangential distortion an be safely ignored If not, we an simply inlude the tangential distortion terms in Eq (2), and the subsequent alibration proedure remains unhanged In the literature, many authors have assumed the model in the form of r = 1 θ + 2 θ θ 5 + [6, 7, 12, 11] As we see shortly, the value of θ for a speified r is neessary in eah step of the optimization iterations Many authors omputed θ by solving a polynomial equation by a numerial means [6, 7, 12, 11], but this auses loss of auray and effiieny It is more onvenient to assume the expansion of θ in terms of r from the beginning From Eq (2), we an express θ in terms of r in the form θ = 2 tan 1( 2f + a 1 ) 3 + a2 ) 5 + )) (3) 3 Inident ray vetor and its derivatives Let m be the unit vetor in the diretion of the inident ray of light whih fouses at (x, y) on the image plane (Fig 1) We all m the inident ray vetor In polar oordinates, it has the expression sin θ os φ m = sin θ sin φ, (4) os θ where θ is the inidene angle from the Z-axis and φ is the azimuthal angle from the X-axis If the prinipal point is at (u 0, v 0 ), we have x u 0 = r os φ, y v 0 = r sin φ, r = (x u 0 ) 2 + (y v 0 ) 2 (5) Hene, Eq (4) is rewritten as m = ((x u 0)/r) sin θ ((y v 0 )/r) sin θ (6) os θ Differentiating this with respet to u 0 and v 0, we obtain m = sin θ u 0 r i, m = sin θ j, (7) v 0 r where we define i (1, 0, 0) and j (0, 1, 0) Next, we onsider derivation with respet to f Differentiating Eq (2) with respet to f on both sides, we have 0 = 2 tan θ 2 + 2f 1 2 os 2 (θ/2) Hene, we obtain θ ( 2 f = tan θ )( f0 θ ) 2 f os2 2 θ f (8) = 2 f sin θ 2 os θ 2 = 1 sin θ (9) f If follows that the derivative of Eq (6) with respet to f is m f = ((x u 0)/r) os θ ((y v 0 )/r) os θ θ f sin θ = 1 f sin θ ((x u 0)/r) os θ ((y v 0 )/r) os θ sin θ (10) Finally, we onsider derivation with respet to a k Differentiating (2) with respet to a k on both sides, we have ) 2k+1 2f 1 θ = 2 os 2 (11) (θ/2) a k IS3-04-2

3 17th Symp Sensing via Image Information, June 2011, Yokohama, Japan n κ O m 1 m N Figure 2 The inident ray vetors m α of ollinear points p 1,, p N are oplanar Hene, we obtain p 1 θ = f ( 0 r ) 2k+1 os 2 θ a k f 2 (12) If follows that the derivative of Eq (6) with respet to a k is m = ((x u 0)/r) os θ ((y v 0 )/r) os θ θ a k a sin θ k = f ( 0 r ) 2k+1 os 2 θ f 2 p N ((x u 0)/r) os θ ((y v 0 )/r) os θ sin θ (13) All the ost funtions in the subsequent optimization are expressed in terms of the inident ray vetor m Hene, we an ompute derivatives of any ost funtion with respet to any parameter simply by ombining the above derivative expressions, 4 Collinearity onstraint Suppose we observe a ollinear point sequene S κ (the subsript κ enumerates all existing sequenes) onsisting of N points p 1,, p N, and let m 1,, m N be their inident ray vetors If the amera is preisely alibrated, the omputed inident ray vetors should be oplanar (Fig 2) and satisfy (n κ, m α ) = 0, α = 1,, N In the following, we denote the inner produt of vetors a and b by (a, b) If the alibration is inomplete, (n κ, m α ) may not be stritly zero So, we adjust the parameters by imizing (n κ, m α ) 2 = n κ m α m α n κ α S κ α S κ = (n κ, m α m α n κ ) = (n κ, M (κ) n κ ), (14) α S κ where we define M (κ) = α S κ m α m α (15) Equation (14) is a quadrati form of M (κ), so its imimum equals the smallest eigenvalue of M (κ) To enfore the ollinearity onstraint for all ollinear point squenes S κ, we detere the paremeters so as to imize J 1 = (16) all κ 41 First derivatives We onsider first derivatives of with respet to, whih represnts the alibration parameters u 0, v 0, f, a 1, a 2, Differentiating the defining equation M (κ) n κ = n κ (17) with respet to on both sides, we have M (κ) n κ +M (κ) n κ = λ(κ) n κ + n κ (18) Computing the inner produt with n κ on both sides, we obtain (n κ, = λ(κ) M (κ) n κ ) + (n κ, M (κ) n κ ) (n κ, n κ ) + (n κ, n κ ) (19) Sine n κ is a unit vetor, we have (n κ, n κ ) = n κ 2 = 1 Variations of a unit vetor should be orthogonal to itself, so (n κ, n κ /) = 0 Sine M (κ) is a symmetri matrix, we have (n κ, M (κ) n κ /) = (M (κ) n κ, n κ /) = (n κ, n κ /) = 0 Thus, Eq (19) implies = (n κ, M (κ) n κ ) (20) This result is well known as the perturabation theorem of eigenvalue problems [5] From the definiton of M (κ) in Eq (45), we see that M (κ) = N ( mα ( mα ) ) m α + m α N = 2S[ m α m α ] M (κ), (21) where S[ ] denotes symmetrization (S[A] = (A + A )/2) Thus, the first derivatives of the funtion J 1 with respet to = u 0, v 0, f, a 1, a 2, are given as follows: 42 Seond derivatives J 1 = (n κ, M (κ) n κ ) (22) all κ Differentiating Eq (20) with respet to (= u 0, v 0, f, a 1, a 2, ), we obtain 2 = ( n κ, M (κ) n κ ) + (n κ, 2 M (κ) n κ ) +(n κ, M (κ) n κ ) = (n κ, 2 M (κ) n κ ) + 2( n κ, M (κ) n κ ) (23) First, onsider the first term Eq (21) with respet to is 2 M (κ) = 2S[ N ( 2 m α m α + m α Differentiation of ( mα ) ) ], (24) IS3-04-3

4 17th Symp Sensing via Image Information, June 2011, Yokohama, Japan and hene we have (n κ, 2 M (κ) n κ ) = 2 N ( (n κ, 2 m α )(m α, n κ ) +(n κ, m ) α )( m α, n κ ) (25) If the alibration is omplete, we should have (m α, n κ ) = 0 In the ourse of the optimization, we an expet that (m α, n κ ) 0 Hene, Eq (25) an be approximated by (n κ, 2 M (κ) n κ ) 2 M (κ) N (n κ, m α )( m α, n κ ) = 2(n κ, M (κ) n κ), (26) N ( mα )( mα ) (27) This is a sort of the Gauss-Newton approximation Next, onsider the seond term of Eq (23) Sine n κ is a unit vetor, its variations are orthogonal to itself Let 1 2 be the eigenvalues of M (κ) with n κ1, n κ2, and n κ the orresponding unit eigenvetors Sine the eigenvetors of a symmetri matrix are mutually orthogonal, any vetor orthogonal to n κ is expressed as a linear ombination of n κ1 and n κ2 Hene, we an write n κ = β 1n κ1 + β 2 n κ2, (28) for some β 1 and β 2 Substitution of Eqs (20) and (28) into Eq (18) results in M (κ) n κ + M (κ) (β 1 n κ1 + β 2 n κ2 ) = (n κ, M (κ) n κ )n κ + (β 1n κ1 + β 2 n κ2 ) (29) Noting that M (κ) n κ1 2 n κ2, we have = 1 n κ1 and M (κ) n κ2 = β 1 ( 1 )n κ1 + β 2 ( 2 )n κ2 = (n κ, M (κ) n κ )n κ M (κ) n κ (30) Computing the inner produt with n κ1 and n κ2 on both sides, we obtain β 1 ( 1 ) = (n κ1, M (κ) n κ ), β 2 ( 2 ) = (n κ2, M (κ) n κ ) (31) Thus, Eq (28) is written as follows: n κ = (n κ1, M (κ) n κ )n κ1 1 (n κ2, M (κ) 2 n κ )n κ2 (32) This is also a well known result of the perturbation theorem of eigenvalue problems [5] Thus, the seond term of Eq (23) an be written as 2( n κ, M (κ) n κ ) = 2(n κ1, M (κ) n κ )(n κ1, M (κ) n κ) 1 2(n κ2, M (κ) n κ )(n κ2, M (κ) n κ) 2 (33) l g n κ O Figure 3 The surfae normals n κ to the planes defined by parallel lines are orthogonal to the ommon diretion l g of the lines Combining Eq (26) and (33), we an approximate Eq (23) in the form 2 ( 2 (n κ, M (κ) n κ) 2 (n κi, M (κ) n κ )(n κi, M (κ) n ) κ) (34) i=1 i Thus, the seond derivatives of the funtion J 1 with respet to and are given by 2 J 1 = 2 ( (n κ, M (κ) n κ) all κ 2 (n κi, M (κ) n κ )(n κi, M (κ) n ) κ) (35) i=1 i 5 Parallelism onstraint Let G g be a group of parallel ollinear point sequenes (the subsript g enumerates all existing groups) with a ommon orientation l g (unit vetor) The normals n κ to the planes passing through the origin O (lens enter) and lines of G g are all orthogonal to l g (Fig 3) Hene, we should have (l g, n κ ) = 0, κ G g, if the alibration is omplete So, we adjust the parameters by imizing where we define κ G g (l g, n κ ) 2 = l g κ G g l g n κ n κ l g = (l g, κ G g n κ n κ l g ) = (l g, N (g) l g ), (36) N (g) = κ G g n κ n κ (37) Equation (36) is a quadrati form of N (g), so its imum equals the smallest eigenvalue µ (g) of N (g) To enfore the parallelism onstraint for all groups of parallel ollinear sequenes, we detere the parameters so as to imize J 2 = all g µ (g) (38) IS3-04-4

5 17th Symp Sensing via Image Information, June 2011, Yokohama, Japan l g l g Figure 4 If two sets of parallel lines make right angles, their diretions l g and l g are orthogonal to eah other 61 First derivatives The first derivatives of the funtion J 3 with respet to parameters are given by J 3 = 2 all orthogonal pairs {G g, G g} ( (l g, l g ) ( l g, l g ) + (l g, The first derivative l g / is given by l g ) ) (43) 51 First derivatives Doing the same perturbation analysis as in Se 4, we obtain the first derivatives of the funtion J 2 with respet to parameters in the form J 2 = (l g, N (g) l g ), all g N (g) = 2S[ κ G g n κ n κ ], (39) where l g is the unit eigenvetor of the matrix N (g) in Eq (37) for the smallest eigenvalue µ (g) The first derivative of n κ / is given by Eq (32) 52 Seond derivatives Doing the same perturbation analysis as in Se 4, we obtain the seond derivatives of the funtion J 2 with respet to parameters and in the form 2 J 2 = 2 ( (l g, N (g) l g) all g 2 i=1 (l gi, N (g) l g )(l gi, N (g) l g) µ (g) i µ (g) ), (40) where µ (g) i, i = 1, 2, are the first and the seond largest eigenvalues of the matrix N (g) and l gi are the orresponding unit eigenvetors The matrix N (g) is defined by N (g) ( nκ )( nκ ) (41) κ G g 6 Orthogonality onstraint Suppose we observe two groups G g and G g of parallel line sequenes with mutually orthogonal diretions l g and l g (Fig 4) The orientation l g of the sequenes in the group G g is the unit eigenvetor of the matrix N (g) in Eq (37) for the smallest eigenvalue If the alibration is omplete, we should have (l g, l g ) = 0, so we adjust the parameters by imizing J 3 = (l g, l g ) 2 (42) all orthogonal pairs {G g, G g} l g 2 = and l g / similarly i=1 62 Seond derivatives (l gi, N (g) µ (g) i l g )l gi, (44) µ (g) Using the Gauss-Newton approximation (l g, l g ) 0, we obtain the seond derivatives of the funtion J 3 with respet to parameters and in the form 2 J 3 = 2 ( ( l g, l g ) + (l g, all orthogonal pairs {G g, G g} l g ) ) ( ( l g, l g ) + (l l ) g g, ) (45) 7 Levenberg-Marquardt proedure To inorporate all of the ollinearity, parallelism, and orthogonality onstraints, we imize J = J 1 γ 1 + J 2 γ 1 + J 3 γ 1, (46) where γ i, i = 1, 2, 3, are the weights to balane the magnitudes of the three terms Note that J 1 J 2 J 3, sine J 1 is proportional to the number of all points, J 2 to the number of all lines, and J 3 to the number of orthogonal pairs of parallel lines In our experiment, we used as γ i the initial value of J i Sine we have derived the first and seond derivatives of all J i with respet to all the parameters, we an now ombine them into the following Levenberg- Marquardt (LM) proedure [15]: 1 Provide initial values, eg, let the prinipal point (u 0, v 0 ) be at the frame enter, f be a default value and, let a 1 = a 2 = = 0 Let J 0 be the value of the funtion J for these initial values, and let C = Compute the inidene angle θ κα of the αth point p α in the κth sequene S κ by Eq (3), and ompute its inidene ray vetor m κα by Eq (6) Then, ompute the derivatives m κα / by Eqs (7), (10), and (13), = u 0, v 0, f, a 1, a 2, 3 Compute the first derivatives J and the seond derivatives J of the funtion J,, = u 0, v 0, f, a 1, a 2, IS3-04-5

6 17th Symp Sensing via Image Information, June 2011, Yokohama, Japan (a) (b) Figure 5 Stripe patterns in four diretions Figure 6 (a) Fisheye lens image of a stripe pattern (b) Deteted edges 4 Detere the inrements u 0, v 0, f, a 1, by solving the linear equation 0 1 (1+C)J u0 u 0 J u0 v 0 J u0 f J u0 a 1 J v0 u 0 (1+C)J v0 v 0 J v0 f J v0 a 1 J fu0 J fv0 (1+C)J ff J fa1 B J a1 0 J a1 v 0 J a1 f (1 + C)J a1 a 1 C A u 0 J u0 v 0 J v0 f J = f B 1 C B C (47) A J a1 5 Tentatively update the parameter values in the form ũ 0 = u 0 + u 0, ṽ = v 0 + v 0, f = f + f, ã 1 = a 1 + a 1, ã 2 = a 2 + a 2, (48) and evaluate the resulting value J of the funtion J 6 If J < J0, proeed Else, let C 10C and go bak to Step 4 7 Let u 0 ũ 0, v 0 ṽ 0, f f, a 1 ã 1, a 2 ã 2, If u 0 < ɛ 0, v 0 < ɛ 0, f < ɛ f, a 1 < ɛ 1, a 2 < ɛ 2,, return J, u 0, v 0, f, a 1, a 2,, and stop Else, let J 0 J and C C/10, and go bak to Step 2 8 Experiments Figure 5 shows the four stripe patterns we used in our experiments We displayed them on a large video sreen and take their images by plaing the amera in various positions so that the pattern image appears in various parts of the view (reall that the view annot be overed by a single planar pattern image) The four patterns were repeatedly displayed ylially with blank frames in-between, and the amera is fixed in eah position at least for one round of the display to take images of the four types Figure 6(a) is one shot of suh images The image size is pixels From eah image, we deteted edges; Fig 6(b) shows the edges deteted from the image in Fig 6(a) We manually removed edges outside the display area We also removed too small lusters of edge points Then, we ran an edge segmentation algorithm to lassify the remaining edge points into onneted edge segments On eah segment was imposed the ollinearity onstraint; on the segments resulting from one stripe pattern were imposed the parallelism onstraint; on the segments resulting from onseutive stripe patterns for a fixed amera position were imposed the orthogonality onstraint In all, we obtained 220 segments, onsisting of 20 groups of parallel segments and 10 orthogonal pairs, to whih the LM proedure was applied Let us all the number K of the terms on the left hand side of Eq (2) the orretion degree, meaning that the left-hand side of Eq (2) is approximated by a (2K + 1)th degree polynomial The results up to the fifth orretion degree are shown in Table 1 We set the frame enter to be (0, 0) to speify the prinipal point (u 0, v 0 ) For the onvergene thresholds in Step 7 of the LM iterations, we let ɛ 0 = ɛ f = 10 3, ɛ 1 = 10 5, ɛ 2 = 10 6, ɛ 3 = 10 7, ɛ 4 = 10 8, and ɛ 5 = 10 9 Using various different initial values, we onfirmed that the LM always onverges to the same solution after at most 10 iterations Figure 7(a) plots the graph of Eq (3) for different orretion degrees For the onveniene of appliations, we numerially onverted Eq (3) to express the angle θ in terms of the distane r As we see, the stereographi projetion model in Eq (1) holds fairly well even without any orretion terms (degree 0) The result is almost unhanged for the degrees 3, 4, and 5 (ie, inluding powers up to r 7, r 9, and r 11 ) Thus, there is no need to inrease the orretion terms any further For omparison, Fig 7(b) shows the same result using ollinearity alone; Fig 7() shows the result ollinearity and parallelism In both ases, the graph approahes, as the degree inreases, some r-θ relationship quite different from the stereographi projetion In order to see what this means, we did a retifiation experiment Figure 8(a) shows a fisheye lens image viewing a square grid pattern in approximately 30 degree diretion, and Fig 8(b) is the retified perspetive image, using the parameters of orretion degree 5 in Table 1 The image is onverted to a view as if observed by rotating the amera by 60 degrees to fae the pattern (see Appendix for this omputation) The blak area near the left boundary orresponds to 95 degrees or more from the optial axis Thus, we an obtain a orret perspetive image to the very boundary of the view IS3-04-6

7 17th Symp Sensing via Image Information, June 2011, Yokohama, Japan Table 1 degree u0 v0 f a1 /10 2 a2 /10 3 a3 /10 4 a4 /10 5 a5 / Computed parameters for eah orretion degree (a) (b) () Figure 7 The dependene of the distane r (pixels) from the foal point on the inidene angle θ (degree) obtained by (a) using the ollinearity, parallelism, and orthogonality onstraints; (b) using only the ollinearity onstraints; () using the ollinearity and parallelism onstraints (a) (b) () Figure 8 (a) Fisheye lens image viewing a square grid pattern in approximately 30 degree diretion (b) Retified perspetive image to be observed if the amera is rotated by 60 degrees to fae the pattern () Similarly retified image using a spurious solution For omparison, Fig 8() shows the result obtained by the same proedure using the spurious solution We an see that ollinear points are ertainly retified to be ollinear and parallel lines to be (a skewed view of) parallel lines The existene of suh a spurious solution has not been known in the past ollinearitybased work [1, 7, 11, 12, 14, 16] Spurious solutions may be prevented by using auxiliary information suh as vanishing point estimation [2, 4, 10, 11] or antipodal point extration [13] As we see, however, the spurious solution does not arise if the orthogonality onstraint is imposed in addition The top-left of Fig 9 is an image of a street sene taken from a moving vehile with a fisheye lens amera mounted below the bumper at the ar front The top-right of Fig 9 is the retified perspetive images The seond and third rows of Fig 9 show the retified perspetive images to be observed if the amera is rotated by 90 left, right, up and down, onfirg that we are really seeing more than 180 angles of view Using a fisheye lens amera like this, we an detet vehiles approahing from left and/or right or reate an image as if viewing the road from above the ar 9 Conluding remarks We have presented a new tehnique for alibrating ultra-wide fisheye lens ameras The basi priniple of our alibration is the imposition of the onstraint that ollinear points be retified to ollinear, parallel lines to parallel, and orthogonal lines to be orthogonal Exploiting the fat that line fitting redues to eigenvalue problems, we optimized the onstraint by invoking the perturbation theorem, as suggested by Onodera and Kanatani [14] in 1992 Then, we derived a Levenberg-Marquardt proedure for it By experiments, we have found that a spurious solution exists if the ollinearity onstraint alone is used or even ombined with the parallelism onstraint However, we have shown that by inorporating the orthogonality onstraint an aurate alibration is done without using any auxiliary information suh as vanishing point estimation [2, 4, 10, 11] Finally, we have shown a real image example using a vehile-mounted fisheye lens amera Aknowledgments The authors thank Ryota Moriyasu of ERD Corporation for helping the experiments This IS3-04-7

8 17th Symp Sensing via Image Information, June 2011, Yokohama, Japan [11] [12] Original fisheye lens image Retified front view [13] [14] Retified left 90 view [15] Retified right 90 view [16] [17] Retified down view IEICE Trans Inf & Syst, J89-D-II-9 (2005-9), M Nakano, S Li and N Chiba, Calibrating fisheye amera by stripe pattern based upon spherial model (in Japanese),, IEICE Trans Inf & Syst, J89-D-1 (2007-1), R Okutsu, K Terabayashi, Y Aragaki, N Shimomura, and K Umeda, Generation of overhead view images by estimating intrinsi and extrinsi amera parameters of multiple fish-eye ameras, Pro IAPR Conf Mahine Vision Appliations, May 2009, Yokohama, Japan, pp R Okutsu, K Terabayashi and K Umeda, Calibration of intrinsi parameters of a fish-eye amera using a sphere (in Japanese), IEICE Trans Inf & Syst, J89-D-12 (201012), Y Onodera and K Kanatani, Geometri orretion of images without amera registration (in Japanese), IEICE Trans Inf & Syst, J75-D-II-5 (1992-5), W H Press, S A Teukolsky, W T Vetterling, and B P Flannery, Numerial Reipes in C: The Art of Sientifi Computing, 2nd ed, Cambridge University Press, Cambridge, UK, 1992 F Swaathan and S K Nayar, Nonmetri alibration of wide-angle lenses and polyameras, IEEE Trans Patt Anal Mah Intell, ( ), Z Zhang, Flexible new tehnique for amera alibration, IEEE Trans Patt Anal Mah Intell, ( ), Retified up view Figure 9 Fisheye lens image of an outdoor sene taken from a moving vehile, retified front images, and retified images after virtually rotating the amera to left, right, up, and down work was supported in part by the Ministry of Eduation, Culture, Sports, Siene, and Tehnology, Japan, under a Grant in Aid for Sientifi Researh (C ) Referenes [1] F Devernay and O Faugeras, Straight lines have to be straight: Automati alibration and removal of distortion from senes of strutured environments, Mahine Vision Appl, 13-1 (2001-8), [2] R Hartley and S B Kang, Parameter-free radial distortion orretion with enter of distortion estimation, IEEE Trans Patt Anal Mah Intell, 28-8 (2007-8), [3] J Heikkila, Geometri amera alibration using irular ontrol points, IEEE Trans Patt Anal Mah Intell, 2210 ( ) [4] C Hughes, P Denny, M Glavin and E Jones, Equidistant fish-eye alibration and retifiation by vanishing point extration, IEEE Trans Patt Anal Mah Intell, ( ), [5] K Kanatani, Statistial Optimization for Geometri Computation: Theory and Pratie, Elsevier, Amsterdam, the Netherlands, 1996; reprinted Dover, New York, NY, USA, 2005 [6] J Kannala and S S Brandt, A general amera model and alibration method for onventional, wide angle, and fisheye-lenses, IEEE Trans Patt Anal Mah Intell, 28-8 (2006-8), [7] S Kase, H Mitsumoto, Y Aragaki, N Shimomura and K Umeda, A method to onstrut overhead view images using multiple fish-eye ameras (in Japanese),, J JSPE, 75-2 (2009-2), [8] H Komagata, I Ishii, A Takahashi, D Wakabayashi and H Imai, A geometri alibration method of internal amera parameters for fish-eye lenses (in Japanese), IEICE Trans Inf & Syst, J89-D-1 (2006-1), [9] Y-C Liu, K-Y Lin, and Y-S Chen, Bird s eye view vision system for vehile surrounding monitoring, Pro 2nd Int Workshop, Robvis2008, February 2008, Aukland, New Zealand, pp [10] M Nakano, S Li and N Chiba, Calibration of fish-eye amera for aquisition of spherial image (in Japanese), Appendix: Image retifiation The retifiation to a perspetive image is done by the following proedure: 1 For eah pixel (x, y ), ompute the inident angle θ by X2 + Y 2 R2 Z = tan, (49) θ = tan Z Z and ompute the orresponding 3-D point (X, Y, Z) by x X y A= p (50) x 2 + y 2 + f 2 f Z where the radius R is arbitrary (we may let R = 1) 2 Compute the orresponding pixel position (x, y) by «««r(θ) X x u0, (51) = + y v0 R2 Z 2 Y and opy its pixel value to (x, y ) If (x, y) are not integers, its value is interpolated from surrounding pixels To generate a perspetive image to be observed by rotating the amera by R (rotation matrix) is obtained if Eq (50) is replaed by x X A= p y A (52) x 2 + y 2 + f 2 f Z IS3-04-8