Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization

Transcription

1 Memoirs of the Faulty of Engineering, Okayama University, Vol 46, pp 10 20, January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization Kenihi KANATANI Department of Computer Siene, Okayama University Okayama Japan Reeived November 21, 2011) We present a new tehnique for alibrating ultra-wide fisheye lens ameras by imposing the onstraint that ollinear points be retified to be ollinear, parallel lines to be parallel, and orthogonal lines to be orthogonal Exploiting the fat that line fitting redues to an eigenvalue problem, we do a rigorous perturbation analysis to obtain a Levenberg-Marquardt proedure for the optimization Doing experiments, we point out that spurious solutions exist if ollinearity and parallelism alone are imposed Our tehnique has many desirable properties For example, no metri information is required about the referene pattern or the amera position, and separate stripe patterns an be displayed on a video sreen to generate a virtual grid, eliating the grid point extration proessing 1 Introdution Fisheye lens ameras are widely used for surveillane purposes beause of their wide angles of view They are also mounted on vehiles for various purposes inluding obstale detetion, self-loalization, and bird s eye view generation [10, 13] However, fisheye lens images have a large distortion, so that in order to apply the omputer vision tehniques aumulated for the past deades, one first needs to retify the image into a perspetive view Already, there is a lot of literature for this [2, 5, 7, 8, 11, 12, 13, 14, 17, 18] The standard approah is to plae a referene grid plane and math the image with the referene, whose preise geometry is assumed to be known [2, 4, 5, 7, 18] However, this approah is not very pratial for reently popularized ultra-wide fisheye lenses, beause they over more than 180 angles of view and hene any even infinite) referene plane annot over the entire field of view This diffiulty an be irumvented by using the ollinearity onstraint pointed out repeatedly, first by Onodera and Kanatani [15] in 1992, later by Swaathan and Nayar [17] in 2000, and by Devernay and Faugeras [1] in 2001 They pointed out that amera alibration an be done by imposing the onstraint that straight lines should be retified to be straight This priniple was applied to fisheye lenses by Nakano, et al [12], Kase kanatani@surisokayama-uajp et al [8], and Okutsu et al [13] Komagata et al [9] further introdued the parallelism onstraint and the orthogonality onstraint, requiring that parallel lines be retified to be parallel and orthogonal lines to be orthogonal However, the ost funtion has been diretly imized by brute fore means suh as the rent method and the Powell method [16] In this paper, we adopt the ollinearityparallelism-orthogonality onstraint of Komagata et al [9] and optimize it by eigenvalue imization The fat that imposing ollinearity implies eigenvalue imization and that the optimization an be done by invoking the perturbation theorem was pointed out by Onodera and Kanatani [15] Using this priniple, they retified perspetive images by gradient deent Here, we onsider ultra-wide fisheye lenses and do a more detailed perturbation analysis to derive the Levenberg-Marquardt proedure, urrently regarded as the standard tool for effiient and aurate optimization The eigenvalue imization priniple has not been known in the past ollinearity-based work [1, 8, 12, 13, 17] Demonstrating its usefulness for ultrawide fisheye lens alibration is the first ontribution of this paper We also show by experiments that the orthogonality onstraint plays an essential role, pointing out that a spurious solution exists if only ollinearity and parallelism are imposed This fat has not been known in the past ollinearity-based work [1, 8, 12, 13, 15, 17] Pointing this out is the This work is subjeted to opyright All rights are reserved by this author/authors 10

2 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization seond ontribution of this paper For data aquisition, we take images of stripes of different orientations on a large-sreen display by plaing the amera in various positions Like the past ollinearity-based methods [1, 8, 12, 13, 15, 17], our method is non-metri in the sense that no metri information is required about the amera position or the referene pattern Yet, many researhers pointed out the neessity of some auxiliary information For example, Nakano et al [11, 12] proposed vanishing point estimation using oni fitting to straight line images reently, Hughes et al [5] proposed this same tehnique again) Okutsu et al [14] piked out the images of antipodal points by hand Suh auxiliary information may be useful to suppress spurious solutions However, we show that aurate alibration is possible without any auxiliary information by our eigenvalue imization for the ollinearityparallelism-orthogonality onstraint This paper is organized as follows In Se 2, we desribe our imaging geometry model Setion 3 gives derivative expressions of the fundamental quantities, followed by a detailed perturbation analysis of the ollinearity onstraint in Se 4, of the parallelism onstraint in Se 5, and of the orthogonality onstraint in Se 6 Setion 7 desribes our Levenberg- Marquardt proedure for eigenvalue imization In Se 8, we experiment our non-metri tehnique, using stripe images on a video display We point out that a spurious solution arises if only ollinearity and parallelism are imposed and that it an be eliated without using any auxiliary information if orthogonality is introdued We also show some examples of real sene appliations In Se 9, we onlude 2 Geometry of Fisheye Lens Imaging We onsider reently popularized ultra-wide fisheye lenses with the imaging geometry modeled by the stereographi projetion r = 2f tan θ 2, 1) where θ is the inidene angle the angle of the inident ray of light from the optial axis) and r in pixels) is the distane of the orresponding image point from the prinipal point Fig 1) The onstant f is alled the foal length We onsider 1) merely beause our amera is as suh, but the following alibration proedure is idential whatever model is used For a manufatured lens, the value of f is unknown or may not be exat if provided by the manufaturer Also, the prinipal point may not be at the enter of the image frame Moreover, 1) is an idealization, and a real lens may not exatly satisfy it So, we generalize 1) into the form r f 0 + a 1 r f 0 ) 3 + a2 r f 0 ) 5 + = 2f f 0 tan θ 2, 2) r Figure 1: The imaging geometry of a fisheye lens and the inident ray vetor m and detere the values of f, a 1, a 2, along with the prinipal point position Here, f 0 is a sale onstant to keep the powers r k within a reasonable numerial range in our experiment, we used the value f 0 = 150 pixels) The linear term r/f 0 has no oeffiient beause f on the right-hand side is an unknown parameter; a 1 = a 2 = = 0 orresponds to the stereographi projetion Sine a suffiient number of orretion terms ould approximate any funtion, the right-hand side of 2) ould be any funtion of θ, eg, the perspetive projetion model f/f 0 ) tan θ or the equidistane projetion model f/f 0 )θ We adopt the stereographi projetion model merely for the ease of initialization In 2), even power terms do not exist, beause the lens has irular symmetry; r is an odd funtion of θ We assume that the azimuthal angle of the projetion is equal to that of the inident ray In the past, these two were often assumed to be slightly different, and geometri orretion of the resulting tangential distortion was studied Currently, the lens manufaturing tehnology is highly advaned so that the tangential distortion an be safely ignored If not, we an simply inlude the tangential distortion terms in 2), and the subsequent alibration proedure remains unhanged In the literature, the model of the form r = 1 θ + 2 θ θ 5 + is frequently assumed [7, 8, 13, 12] As we see shortly, however, the value of θ for a speified r is neessary in eah step of the optimization iterations So, many authors omputed θ by solving a polynomial equation using a numerial means [7, 8, 13, 12], but this auses loss of auray and effiieny It is more onvenient to express θ in terms of r from the beginning From 2), the expression of θ is given by θ = 2 tan 1 f 0 r r ) 3 r ) 5 + a 1 + a2 + )) 2f f 0 f 0 f 0 3) 3 Inident Ray Vetor Let m be the unit vetor in the diretion of the inident ray of light Fig 1); we all m the inident θ f m 11

3 Kenihi KANATANI MEMFACENGOKAUNI Vol 46 ray vetor In polar oordinates, it has the expression sin θ os φ m = sin θ sin φ, 4) os θ where θ is the inidene angle from the -axis and φ is the azimuthal angle from the X-axis Sine φ, by our assumption, equals the azimuthal angle on the image plane, the point x, y) on whih the inident light fouses is speified by x u 0 = r os φ, y v 0 = r sin φ, r = x u 0 ) 2 + y v 0 ) 2, 5) where u 0, v 0 ) is the prinipal point Hene, 4) is rewritten as x u 0 )/r) sin θ m = y v 0 )/r) sin θ 6) os θ Differentiating 5) with respet to u 0 and v 0, we obtain r = x u 0, u 0 r r = y v 0 7) v 0 r Hene, the derivatives of 6) with respet to u 0 and v 0 beome m = 1/r + x u 0) 2 /r 3 x u 0 )y v 0 )/r 3 sin θ u 0 0 x u 0)/r) os θ θ + y v 0 )/r) os θ sin θ m = x u 0)y v 0 )/r 3 1/r + y v 0 ) 2 /r 3 v x u 0)/r) os θ y v 0 )/r) os θ sin θ, u 0 sin θ θ v 0 8) Differentiating 3) with respet to u 0 and v 0 on both sides, we obtain 1 + 3a 1 r ) 2 5a 2 r ) 4 7a 3 r ) 6 r ) f 0 f 0 f 0 f 0 f 0 f 0 f 0 u 0 = 2f 1 θ f 0 2 os 2, θ/2) u a 1 r ) 2 5a 2 r ) 4 7a 3 r ) 6 r ) f 0 f 0 f 0 f 0 f 0 f 0 f 0 v 0 = 2f 1 θ f 0 2 os 2 9) θ/2) v 0 Hene, θ/ u 0 and θ/ v 0 an be written as θ = 1 θ u 0 f os r ) 2k ) x u0 2k 1)a k, f 0 r k=1 θ = 1 θ v 0 f os r ) 2k ) y v0 2k 1)a k f 0 r k=1 10) Substituting these into 9), we an evaluate m/ u 0 and m/ v 0 Next, we onsider derivation with respet to f Differentiating 2) with respet to f on both sides, we have Hene, we obtain 0 = 2 tan θ f f 1 f 0 2 os 2 θ/2) θ f 11) θ f = 2 f sin θ 2 os θ 2 = 1 sin θ 12) f It follows that the derivative of 6) with respet to f is m f = 1 f sin θ x u 0)/r) os θ y v 0 )/r) os θ 13) sin θ Finally, we onsider derivation with respet to a k Differentiating 2) with respet to a k on both sides, we have r ) 2k+1 2f 1 θ = f 0 f 0 2 os 2 14) θ/2) a k Hene, we obtain θ = f 0 r ) 2k+1 os 2 θ a k f f ) It follows that the derivation of 6) with respet to a k is m = f 0 r ) 2k+1 os 2 θ x u 0)/r) os θ y v 0 )/r) os θ a k f f 0 2 sin θ 16) All the ost funtions in the subsequent optimization are expressed in terms of the inident ray vetor m Hene, we an ompute derivatives of any ost funtion with respet to any parameter simply by ombining the above derivative expressions 4 Collinearity Constraint Suppose we observe a ollinear point sequene S κ the subsript κ enumerates all existing sequenes) onsisting of N points p 1,, p N, and let m 1,, m N be their inident ray vetors If the amera is preisely alibrated, the omputed inident ray vetors should be oplanar Hene, if n κ is the unit normal to the plane passing through the origin O lens enter) and S κ, we should have n κ, m α ) = 0, α = 1,, N Fig 2) In the following, we denote the inner produt of vetors a and b by a, b) If the alibration 12

4 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization Computing the inner produt with n κ on both sides, we obtain n κ O m 1 m N Figure 2: The inident ray vetors m α of ollinear points p 1,, p N are oplanar is inomplete, however, n κ, m α ) may not be stritly zero So, we adjust the parameters by imizing n κ, m α ) 2 = n κ m α m α n κ α S κ α S κ = n κ, m α m α n κ ) = n κ, M κ) n κ ), 17) α S κ where we define M κ) = p 1 p N α S κ m α m α 18) Sine 17) is a quadrati form of M κ), its imum equals the smallest eigenvalue λ κ) of M κ), n κ being the orresponding unit eigenvetor To enfore the ollinearity onstraint for all ollinear point sequenes S κ, we detere the parameters so as to imize J 1 = all κ λ κ) 19) eause we imize the sum of eigenvalues, we all our approah eigenvalue imization, whih was first proposed by Onodera and Kanatani [15] in 1992 The reason that this tehnique has not been widely used may be that at first sight it appears that one annot differentiate eigenvalues However, differentiating eigenvalue is very easy, as we now show in the following, if one uses the perturbation theorem well known in physis 41 First derivatives Consider the first derivatives of λ κ) with respet to, whih represents the parameters u 0, v 0, f, a 1, a 2, Differentiating the defining equation M κ) n κ = λ κ) n κ 20) with respet to on both sides, we have M κ) n κ +M κ) n κ = λκ) n κ +λ κ) n κ 21) n κ, = λκ) M κ) n κ ) + n κ, M κ) n κ ) n κ, n κ ) + λ κ) n κ, n κ ) 22) Sine n κ is a unit vetor, we have n κ, n κ ) = n κ 2 = 1 Variations of a unit vetor should be orthogonal to itself, so n κ, n κ /) = 0 Sine M κ) is a symmetri matrix, we have n κ, M κ) n κ /) = M κ) n κ, n κ /) = λ κ) n κ, n κ /) = 0 Thus, 22) implies λ κ) = n κ, M κ) n κ ) 23) This result is well known as the perturbation theorem of eigenvalue problems [6] From the definition of M κ) in 49), we see that M κ) = N mα mα ) ) m α + m α N = 2S[ m α m α ] M κ), 24) where S[ ] denotes symmetrization S[A] = A + A )/2) Thus, the first derivatives of the funtion J 1 with respet to = u 0, v 0, f, a 1, a 2, are given as follows: J 1 = n κ, M κ) n κ ) 25) all κ 42 Seond derivatives Differentiating 23) with respet to = u 0, v 0, f, a 1, a 2, ), we obtain 2 λ κ) = n κ, M κ) n κ ) + n κ, 2 M κ) n κ ) +n κ, M κ) n κ ) = n κ, 2 M κ) n κ ) + 2 n κ, M κ) n κ ) 26) First, onsider the first term 24) with respet to is 2 M κ) = 2S[ N 2 m α and hene we have n κ, 2 M κ) n κ ) = 2 m α + m α N Differentiation of mα ) ) ], 27) n κ, 2 m α )m α, n κ ) +n κ, m α ) m α, n κ ) ) 28) 13

5 Kenihi KANATANI MEMFACENGOKAUNI Vol 46 If the alibration is omplete, we should have m α, n κ ) = 0 In the ourse of the optimization, we an expet that m α, n κ ) 0 Hene, 28) an be approximated by n κ, 2 M κ) n κ ) 2 M κ) N n κ, m α ) m α, n κ ) = 2n κ, M κ) n κ), 29) N mα ) mα ) 30) This is a sort of the Gauss-Newton approximation Next, onsider the seond term of 26) eause n κ is a unit vetor, its variations are orthogonal to itself Let λ κ) 1 λ κ) 2 λ κ) be the eigenvalues of M κ) with n κ1, n κ2, and n κ the orresponding unit eigenvetors Sine the eigenvetors of a symmetri matrix are mutually orthogonal, any vetor orthogonal to n κ is expressed as a linear ombination of n κ1 and n κ2 Hene, we an write n κ = β 1n κ1 + β 2 n κ2, 31) for some β 1 and β 2 Substitution of 23) and 31) into 21) results in M κ) n κ + M κ) β 1 n κ1 + β 2 n κ2 ) = n κ, M κ) n κ )n κ + λ κ) β 1n κ1 + β 2 n κ2 ) 32) Noting that M κ) n κ1 λ κ) 2 n κ2, we have = λ κ) 1 n κ1 and M κ) n κ2 = β 1 λ κ) 1 λ κ) )n κ1 + β 2 λ κ) 2 λ κ) )n κ2 = n κ, M κ) n κ )n κ M κ) n κ 33) Computing the inner produt with n κ1 and n κ2 on both sides, we obtain β 1 λ κ) 1 λ κ) ) = n κ1, M κ) n κ ), β 2 λ κ) 2 λ κ) ) = n κ2, M κ) n κ ) 34) Thus, 31) is written as follows: n κ = n κ1, M κ) n κ )n κ1 λ κ) 1 λ κ) n κ2, M κ) λ κ) 2 λ κ) n κ )n κ2 35) This is also a well known result of the perturbation theorem of eigenvalue problems [6] Thus, the seond term of 26) an be written as 2 n κ, M κ) n κ ) = 2n κ1, M κ) n κ )n κ1, M κ) n κ) λ κ) 1 λ κ) 2n κ2, M κ) n κ )n κ2, M κ) n κ) λ κ) 2 λ κ) 36) l g n κ O Figure 3: The surfae normals n κ to the planes defined by parallel lines are orthogonal to the ommon diretion l g of the lines Combining 29) and 36), we an approximate 26) in the form 2 λ κ) 2 n κ, M κ) n κ) 2 n κi, M κ) n κ )n κi, M κ) n ) κ) 37) i=1 λ κ) i λ κ) Thus, the seond derivatives of the funtion J 1 with respet to and are given by n κ, M κ) 2 J 1 = 2 all κ 2 i=1 n κ) n κi, M κ) n κ )n κi, M κ) λ κ) i l λ κ) 5 Parallelism Constraint g n κ) ) 38) Let G g be a group of parallel ollinear point sequenes the subsript g enumerates all existing groups) with a ommon orientation l g unit vetor) The normals n κ to the planes passing through the origin O lens enter) and lines of G g are all orthogonal to l g Fig 3) Hene, we should have l g, n κ ) = 0, κ G g, if the alibration is omplete So, we adjust the parameters by imizing l g, n κ ) 2 = l g n κ n κ l g κ G g κ G g where we define = l g, κ G g n κ n κ l g ) = l g, N g) l g ), 39) N g) = κ G g n κ n κ 40) Sine 39) is a quadrati form of N g), its imum equals the smallest eigenvalue µ g) of N g), l g being the orresponding unit eigenvetor To enfore the parallelism onstraint for all groups of parallel ollinear sequenes, we detere the parameters so as to imize J 2 = all g µ g) 41) 14

6 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization l g 61 First derivatives l g The first derivatives of the funtion J 3 with respet to parameters are given by J 3 = 2 all orthogonal pairs {G g, G g} l g, l g ) l g, l g ) + l g, l g ) ) 47) The first derivative l g / is given by Figure 4: If two sets of parallel lines make right angles, their diretions l g and l g are orthogonal to eah other 51 First derivatives Doing the same perturbation analysis as in Se 4, we obtain the first derivatives of the funtion J 2 with respet to parameters in the form J 2 = l g, N g) l g ), 42) all g N g) where n κ / is given by 35) 52 Seond derivatives = 2S[ κ G g n κ n κ ], 43) Doing the same perturbation analysis as in Se 4, we obtain the seond derivatives of the funtion J 2 with respet to parameters and in the form 2 J 2 = 2 l g, N g) l g) all g N g) 2 i=1 l gi, N g) l g )l gi, N g) l g) µ g) i κ G g nκ µ g) ), 44) ) nκ ), 45) where µ g) i, i = 1, 2, are the first and the seond largest eigenvalues of the matrix N g) and l gi are the orresponding unit eigenvetors 6 Orthogonality Constraint Suppose we observe two groups G g and G g of parallel line sequenes with mutually orthogonal diretions l g and l g Fig 4) The orientation l g of the sequenes in the group G g is the unit eigenvetor of the matrix N g) in 40) for the smallest eigenvalue If the alibration is omplete, we should have l g, l g ) = 0, so we adjust the parameters by imizing J 3 = l g, l g ) 2 46) all orthogonal pairs {G g, G g} l g 2 = and l g / similarly i=1 62 Seond derivatives l gi, N g) µ g) i l g )l gi, 48) µ g) Using the Gauss-Newton approximation l g, l g ) 0, we obtain the seond derivatives of the funtion J 3 with respet to parameters and in the form 2 J 3 = 2 l g, l g ) + l g, all orthogonal pairs {G g, G g} l g ) ) l g, l g ) + l l ) g g, ) 49) 7 Levenberg-Marquardt Proedure To inorporate all of the ollinearity, parallelism, and orthogonality onstraints, we imize J = J 1 γ 1 + J 2 γ 1 + J 3 γ 1, 50) where γ i, i = 1, 2, 3, are the weights to balane the magnitudes of the three terms Note that J 1 J 2 J 3, sine J 1 is proportional to the number of all points, J 2 to the number of all lines, and J 3 to the number of orthogonal pairs of parallel lines In our experiment, we used as γ i the initial value of J i, so that J is initially = 3 Our ollinearityparallelism-orthogonality onstraint effetively fits straight lines to ollinear point sequenes, but unlike the past ollinearity-based work [1, 8, 12, 13, 17] we never expliitly ompute the fitted lines; all the onstraints are expressed in terms of the inident ray vetors m Now that we have derived the first and seond derivatives of all J i with respet to all the parameters, we an ombine them into the following Levenberg- Marquardt LM) proedure [16]: 1 Provide initial values, eg, let the prinipal point u 0, v 0 ) be at the frame enter, f be a default value and, let a 1 = a 2 = = 0 Let J 0 be the value of the funtion J for these initial values, and let C =

7 Kenihi KANATANI MEMFACENGOKAUNI Vol 46 2 Compute the inidene angle θκα of the αth point pα in the κth sequene Sκ by 3), and ompute its inidene ray vetor mκα by 6) Then, ompute the derivatives mκα / by 8), 13), and 16) for = u0, v0, f, a1, a2, 3 Compute the first derivatives J and the seond derivatives J0 of the funtion J for, 0 = u0, v0, f, a1, a2, 4 Detere the inrements u0, v0, f, a1, by solving the linear equation 0 1+C)Ju0 u0 Ju0 v0 Ju0 f J u0 a1 Jv0 u0 1+C)Jv0 v0 J v0 f Jv0 a1 Jf u0 Jf v0 1+C)Jf f J f a1 Ja1 u0 Ja1 v0 Ja1 f 1 + C)Ja Ju0 u0 Jv0 C v0 C C C f C J C C = f C Ja1 C a1 A 1 C C C C C A 51) 5 Tentatively update the parameter values in the form u 0 = u0 + u0, v = v0 + v0, f = f + f, a 1 = a1 + a1, a 2 = a2 + a2, Figure 5: Stripe patterns in four diretions are displayed on a large video sreen 52) and evaluate the resulting value J of the funtion J 6 If J < J0, proeed Else, let C 10C and go bak to Step 4 7 Let u0 u 0, v0 v 0, f f, a1 a 1, a2 a 2, If u0 < ²0, v0 < ²0, f < ²f, a1 < ²1, a2 < ²2,, return J, u0, v0, f, a1, a2,, and stop Else, let J0 J and C C/10, and go bak to Step 2 a) b) Figure 6: a) Fisheye lens image of a stripe pattern b) Deteted edges 8 Experiments to reate onneted edge segments On eah segment was imposed the ollinearity onstraint; on the segments in one frame were imposed the parallelism onstraint; on the segments in one frame and the frame after the next with the same amera position were imposed the orthogonality onstraint In all, we obtained 220 segments, onsisting of 20 groups of parallel segments and 10 orthogonal pairs, to whih the LM proedure was applied The onventional approah using a referene grid board [2, 5, 7, 17, 18] would require preise loalization of grid points in the image, whih is a rather diffiult task Here, all we need to do is detet ontinuous edges We an use a video display instead of a speially designed grid board, beause our method is non-metri: we need no metri information about The four stripe patterns shown in Fig 5 above) were displayed on a large video sreen Fig 5 below) We took images by plaing the amera in various positions so that the stripe pattern appears in various parts of the view reall that the view annot be overed by a single planar pattern image) The four patterns were ylially displayed with blank frames in-between, and the amera is fixed in eah position for at least one yle to apture the four patterns; Fig 6a) shows one shot The image size is pixels From eah image, we deteted edges; Fig 6b) shows the edges deteted from the image in Fig 6a) We manually removed those edges outside the display area We also removed too small lusters of edge points Then, we ran an edge segmentation algorithm 16

8 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization the pattern or the amera positions We do not even know where eah edge point orresponds to in the referene pattern Let us all the number K of the terms on the left hand side of 2) the orretion degree, meaning that the left-hand side of 2) is approximated by a 2K + 1)th degree polynomial The results up to the fifth orretion degree are shown in Table 1 We set the frame enter to be 0, 0) to speify the prinipal point u 0, v 0 ) For the onvergene thresholds in Step 7 of the LM iterations, we let ɛ 0 = ɛ f = 10 3, ɛ 1 = 10 5, ɛ 2 = 10 6, ɛ 3 = 10 7, ɛ 4 = 10 8, and ɛ 5 = 10 9 Using various different initial values, we onfirmed that the LM always onverges to the same solution after around 10 to 20 iterations Figure 7a) plots the graph of 3) for different orretion degrees For the onveniene of the subsequent appliations, we numerially onverted 3) to express the angle θ in terms of the distane r As we see, the stereographi projetion model in 1) holds fairly well even without any orretion terms degree 0) The result is almost unhanged for the degrees 3, 4, and 5, ie, inluding powers up to r 7, r 9, and r 11 Thus, there is no need to inrease the orretion terms any further For omparison, Fig 7b) shows the same result using ollinearity alone, ie, J = J 1 /γ 1 instead of 50); Fig 7) shows the result using ollinearity and parallelism, ie, J = J 1 /γ 1 + J 2 /γ 2 instead of 50) In both ases, the graph approahes, as the degree inreases, some r-θ relationship quite different from the stereographi projetion In order to see what this means, we did a retifiation experiment Figure 8a) shows a fisheye lens image viewing a square grid pattern in approximately 30 degree diretion, and Fig 8b) is the retified perspetive image, using the parameters of orretion degree 5 in Table 1 The image is onverted to a view as if observed by rotating the amera by 60 degrees to fae the pattern see Appendix for this omputation) The blak area near the left boundary orresponds to 95 degrees or more from the optial axis Thus, we an obtain a orret perspetive image to the very boundary of the view For omparison, Fig 8) shows the result obtained by the same proedure using the spurious solution We an see that ollinear points are ertainly retified to be ollinear and parallel lines to be a skewed view of) parallel lines We have onfirmed that this spurious solution is not the result of a loal imum Let us all the parameter values obtained by imposing ollinearity, parallelism, and orthogonality the orret solution The ost funtions J = J 1 /γ 1 and J = J 1 /γ 1 + J 2 /γ 2 are ertainly smaller at the spurious solution than at the orret solution, meaning that the spurious solution is not attributed to the imization algorithm but is inherent in the formulation of the problem itself The fat that spurious solution should exist is easily understandable if one onsiders perspetive ameras If no image distortion exists, the 3-D interpretation of images using wrong amera parameters, suh as the foal length, the prinipal point, the aspet ratio, and the skew angle, is a projetive transformation of the real sene, alled projetive reonstrution [3] Projetive transformations preserve ollinearity but not parallelism or orthogonality Imposing parallelism, we obtain affine reonstrution Further imposing orthogonality, we obtain so alled Eulidean reonstrution Thus, orthogonality is essential for amera alibration; ollinearity and parallelism alone are insuffiient This fat has been overlooked in the past ollinearity-based work [1, 8, 12, 13, 15, 17], partly beause image distortion is so doant for fisheye lens ameras, partly beause spurious solutions an be prevented by using auxiliary information suh as vanishing point estimation [2, 5, 11, 12] or antipodal point extration [14], and partly beause usually a small number, typially three, of olletion terms are retained [1, 8, 12, 13, 15, 17], providing insuffiient degrees of freedom to fall into spurious solution The top-left of Fig 9 is an image of a street sene taken from a moving vehile with a fisheye lens amera mounted below the bumper at the ar front The top-right of Fig 9 is the retified perspetive image The seond and third rows show the retified perspetive images to be observed if the amera is rotated by 90 left, right, up and down, onfirg that we are really seeing more than 180 angles of view Using a fisheye lens amera like this, we an detet vehiles approahing from left and/or right or reate an image as if viewing the road from above the ar 9 Conluding Remarks We have presented a new tehnique for alibrating ultra-wide fisheye lens ameras Our method an be ontrasted to the onventional approah of using a grid board as follows: 1 For ultra-wide fisheye lens ameras with more than 180 degrees of view, the image of any large even infinite) plane annot over the image frame Our method allows us take multiple partial images of the referene by freely moving the amera so that every part of the frame is overed by some referene images 2 The grid-based approah requires detetion of grid points in the image, but aurate proessing of a grid image is rather diffiult In our method, we only need to detet ontinuous edges of a stripe pattern 3 The grid-based approah requires the orrespondene of eah deteted grid point to the loation on the referene This is often diffiult due to the 17

9 Kenihi KANATANI MEMFACENGOKAUNI Vol 46 Table 1: Computed parameters for eah orretion degree degree u v f a 1/ a 2/ a 3 / a 4 / a 5 / a) b) ) Figure 7: The dependene of the distane r pixels) from the foal point on the inidene angle θ degree) obtained by a) using the ollinearity, parallelism, and orthogonality onstraints; b) using only the ollinearity onstraints; ) using the ollinearity and parallelism onstraints periodiity of the grid In our method, we need not know where the deteted edge points orrespond to in the referene 4 In the grid-based approah, one needs to measure the amera position relative to the referene board by a mehanial means or by omputing the homography from image-referene mathing [2, 18] Our method does not require any information about the amera position 5 In the grid-based approah, one needs to reate a referene pattern This is not a trivial task If a pattern is printed out on a sheet of paper and pasted to a planar surfae, wrinkles and reases may arise and the glue may ause uneven deformations of the paper In our method, no metri information is required about the pattern, so we an display it on any video sreen 6 In the grid-based approah, one an usually obtain only one image of the referene pattern from one amera position In our method, we an fix the amera anywhere and freely hange the referene pattern on the video sreen 7 The diffiulty of proessing a grid image with rossings and branhes is irumvented by generating a virtual grid from separate stripe images of different orientations; eah image has no rossings or branhes The basi priniple of our alibration is the imposition of the onstraint that ollinear points be retified to ollinear, parallel lines to parallel, and orthogonal lines to be orthogonal Exploiting the fat that line fitting redues to eigenvalue problems, we optimized the onstraint by invoking the perturbation theorem, as suggested by Onodera and Kanatani [15] Then, we derived a Levenberg-Marquardt proedure for eigenvalue imization y experiments, we have found that a spurious solution exists if the ollinearity onstraint alone is used or even ombined with the parallelism onstraint This fat has not been notied in existing ollinearity-based work [1, 8, 12, 13, 15, 17] However, we have shown that inorporating the orthogonality onstraint allows an aurate alibration without using any auxiliary information suh as vanishing point estimation [2, 5, 11, 12] We have also shown a real image example using a vehile-mounted fisheye lens amera It is expeted that our proedure is going to be a standard tool for fisheye lens amera alibration Define a world XY oordinate system with the origin O at the lens enter and the -axis along the optial axis Fig 1) As far as the amera imaging is onerned, the outside sene an be regarded as if painted inside a sphere of a speified radius R surrounding the lens The angle θ of the inident ray from point X, Y, ) on the sphere is θ = tan 1 X2 + Y 2 = tan 1 R2 2 53) 18

10 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization a) b) ) Figure 8: a) Fisheye lens image viewing a square grid pattern in approximately 30 degree diretion b) Retified perspetive image to be observed if the amera is rotated by 60 degrees to fae the pattern ) Similarly retified image using a spurious solution ρ X, Y, ) θ f X, Y, ) R f x, y) Original fisheye lens image Retified front view r u0,v 0) 0,0) x, y) Figure 10: To the pixel x, y ) of the retified perspetive image of foal length f should be an image of the 3-D point X, Y, ), whih is atually projeted to x, y) on the fisheye lens image of foal length f point X, Y, ) given by Retified left 90 view Retified right 90 view X x = f, Y y = f 55) From Y2 f 2 R2 X2 x 2 + y 2 + f 2 = f f f 2 =, 56) 2 we see that Retified down view Retified up view f R =, 2 x + y 2 + f 2 57) and hene X x R Y = y x 2 + y 2 + f 2 f Figure 9: Fisheye lens image of an outdoor sene taken from a moving vehile, retified front images, and retified images after virtually rotating the amera to left, right, up, and down Let rθ) be the expression of r in terms of θ desribed in Fig 7 The point X, Y, ) is projeted to an image point x, y) suh that 58) Thus, the retifiation an be done as follows Fig 10): 54) 1 For eah pixel x, y ), ompute the 3-D oordinates X, Y, ) in 58), where the radius R is arbitrary we may let R = 1) Suppose we want to obtain a retified perspetive image with foal length f Then, the pixel x, y ) of the retified image should be the projetion of the 3-D 2 Compute the orresponding pixel position x, y) by 54) and opy its pixel value to x, y ) If x, y) are not integers, interpolate its value from surrounding pixels ) ) ) rθ) x u0 X = + y v0 R2 2 Y 19

11 Kenihi KANATANI MEMFACENGOKAUNI Vol 46 We an also obtain the perspetive image to be observed if the amera is rotated by R Sine rotation of the amera by R is equivalent to the rotation of the sene sphere by R 1, the 3-D point X, Y, ) given by 58) on the rotated sphere orresponds to the 3-D point X Y = R x2 + ȳ 2 + f R x 59) ȳ f 2 on the original sphere Its fisheye lens image should be held at the pixel x, y) given by 54) Thus, the mapping proedure goes as follows: 1 For eah pixel x, ȳ), ompute the 3-D oordinates X, Y, ) in 59) 2 Compute the orresponding pixel position x, y) by 54) and opy its pixel value to x, ȳ) Aknowledgments The author thank Ryota Moriyasu of ERD Corporation for helping the experiments This work was supported in part by the Ministry of Eduation, Culture, Sports, Siene, and Tehnology, Japan, under a Grant in Aid for Sientifi Researh C ) Referenes [1] F Devernay and O Faugeras, Straight lines have to be straight: Automati alibration and removal of distortion from senes of strutured environments, Mahine Vision Appliations, vol 13, no 1, pp 14 24, Aug 2001 [2] R Hartley and S Kang, Parameter-free radial distortion orretion with enter of distortion estimation, IEEE Trans Pattern Analysis and Mahine Intelligene, vol 28, no 8, , Aug 2007 [3] R Hartley and O isserman, Multiple View Geometry in Computer Vision, 2nd ed, Cambridge Univ Press, 2004 [4] J Heikkilä, Geometri amera alibration using irular ontrol points, IEEE Trans Pattern Analysis and Mahine Intelligene, vol 22, no 10, Ot 2000 [5] C Hughes, P Denny, M Glavin and E Jones, Equidistant fish-eye alibration and retifiation by vanishing point extration, IEEE Trans Pattern Analysis and Mahine Intelligene, vol 32, no 12, , De 2010 [6] K Kanatani, Statistial Optimization for Geometri Computation: Theory and Pratie, Elsevier, 1996; reprinted Dover, 2005 [7] J Kannala and S S randt, A general amera model and alibration method for onventional, wide angle, and fisheye-lenses, IEEE Trans Pattern Analysis and Mahine Intelligene, vol 28, no 8, , Aug 2006 [8] S Kase, H Mitsumoto, Y Aragaki, N Shimomura and K Umeda, A method to onstrut overhead view images using multiple fisheye ameras in Japanese),, J JSPE, vol 75, no 2, pp , Feb 2009 [9] H Komagata, I Ishii, A Takahashi, D Wakabayashi and H Imai, A geometri alibration method of internal amera parameters for fisheye lenses in Japanese), IEICE Trans Information and Systems, vol J89-D, no ), 64 73, Jan 2006 [10] Y-C Liu, K-Y Lin, and Y-S Chen, ird s eye view vision system for vehile surrounding monitoring, Pro 2nd Int l Workshop, Robvis2008, pp , Feb 2008 [11] M Nakano, S Li and N Chiba, Calibration of fish-eye amera for aquisition of spherial image in Japanese), IEICE Trans Information and Systems, vol J89-D-II, no 9, , Sept 2005 [12] M Nakano, S Li and N Chiba, Calibrating fisheye amera by stripe pattern based upon spherial model in Japanese), IEICE Trans Information and Systems, vol J89-D, no 1, 73 82, Jan 2007 [13] R Okutsu, K Terabayashi, Y Aragaki, N Shimomura, and K Umeda, Generation of overhead view images by estimating intrinsi and extrinsi amera parameters of multiple fish-eye ameras, Pro IAPR Conf Mahine Vision Appliations, pp , May 2009 [14] R Okutsu, K Terabayashi and K Umeda, Calibration of intrinsi parameters of a fish-eye amera using a sphere in Japanese), IEICE Trans Information and Systems, vol J89-D, no ), , De 2010 [15] Y Onodera and K Kanatani, Geometri orretion of images without amera registration in Japanese), IEICE Trans Information and Systems, vol J75-D-II, no 5, , May 1992 [16] W H Press, S A Teukolsky, W T Vetterling, and P Flannery, Numerial Reipes in C: The Art of Sientifi Computing, seond ed, Cambridge Univ Press, 1992 [17] F Swaathan and S K Nayar, Nonmetri alibration of wide-angle lenses and polyameras, IEEE Trans Pattern Analysis and Mahine Intelligene, vol 22, no 10, , Ot 2000 [18] hang, Flexible new tehnique for amera alibration, IEEE Trans Pattern Analysis and Mahine Intelligene, vol 22, no 11, , Nov