Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
The history of image formation • The idea of a camera is linked to how a perceives the world with her eyes y human p • But in the early days we had only vague or incorrect ideas about TSBB09 Image Sensors – What is light – How the eye maps objects in the 3D world to the “image” that we perceive 2014-HT2 2014 HT2 Lecture A I Image Formation F ti • Prior to the camera: the artist/painter TSBB09, Lecture A, Klas Nordberg, LiU 2 13th century Europe Ancient Egypt 3 4 15th century The Renaissance • Early investigations in perspective started alreadyy byy the ancient Greeks ((~500 BC)) and Arab scientists (~1000 AD) • It was not until the 15th century that artists began to use perspective as a basis for their paintings – Lines that are p parallel in 3D should meet at a single point in the image – Brunelleschi (~1415) ( 1415) TSBB09, Lecture A, Klas Nordberg, LiU Christ handing the keys to Saint Peter, by Perugino 1481 5 Camera obscura 6 Camera obscura • Since ancient times it has been known that a brightly g y illuminated scene can be projected to an image Full sized dedicated camera obscura rooms were built in mansions and castles in the 17th and 18th centuries – In a dark room (Latin: camera obscura) – Through a small hole (aperture) – The image becomes rotated 180o TSBB09, Lecture A, Klas Nordberg, LiU 7 From Diderot’s Encyclopedia 1772 8 Camera obscura Camera obscura • Th The fifirstt photograph h t h was taken t k by b a smallll camera obscura in 1826 by Niépce – 8h exposure time! • Today A camera obscura at Melville Garden in Massachusetts around 1880 – Large sized: as tourist attractions –S Small a ssized: ed for o hobby obby p photographs o og ap s TSBB09, Lecture A, Klas Nordberg, LiU 9 10 Laterna Magica History of photography • Devices that can project an image onto a screen have been described since the 16th century (in Europe, possibly earlier elsewhere?) g ((magic g lamp)) • Referred to as laterna magica • We need only – – – – An image painted on a transparent material (glass) A strong light source A lens A suitable screen From the Ars Lucis et Umbrae, 1671 11 • • • • • • • • • • • • • • • • • • • • • • • 1826: Niépce takes the first proper photograph. 8h exposure time! 1839: Daguerre develops the first practical method for photography 1839: Talbot invents the process for taking negative images that can be copied 1839: Herschel invents glass negatives 1861: Maxwell demonstrates color photographs 1878: Muybridge demonstrates moving images 1887: Celluloid film is introduced 1888: Kodak markets its first “easy-to-use” camera 1891: Edison patents his “kinetoscopic camera” 189 L 1895: Lumiére ié B Bros. iinvent the h ““cinématographe” i é h ” 1925: Leica introduces the 35mm film format for still images 1936: Kodachrome color film 1948: Land invents the Polaroid camera 1957 Fi 1957: Firstt di digitized iti d iimage 1959: AGFA introduces the first automatic camera 1969: Boyle and Smith invent the first CCD chip for image capture (based on the ”bubble memory”) 1973: Fairchild Semiconductor markets the first CCD chip (100 × 100 pixels) 1975 B 1975: Bayer att K Kodak: d k fifirstt single i l chip hi color l CCD camera 1981: Sony markets the Mavica, the 1st consumer digital camera. Stores images on a floppy disc 1986: Kodak presents the first megapixel CCD camera 2005: Film based photography company AgfaPhoto files for insolvency 2006 D 2006: Dalsa l C Corporation ti presents t a 111 M Mpixel i l CCD camera 2009: Kodak announces that it will discontinue production of Kodachrome film 12 Source: en.wikipedia.org Basic physics Frequency and wavelength • Electromagnetic radiation consists of g waves electromagnetic • The relation between frequency and g is wavelength – With energy – That propagate through space c=λν • The waves consist of transversal electrical and magnetic fields that alternate with a temporal p frequency q y ν ((Hertz)) and spatial p wavelength λ (meter) TSBB09, Lecture A, Klas Nordberg, LiU 13 Particles and energy Energy increases with ν and decreases with λ • E Energy depends d d on th the ffrequency ν • Energy is preserved c1 ν1 E=hν=hc/λ h is Planck’s constant (≈ 6.623 · 10-34 Js) TSBB09, Lecture A, Klas Nordberg, LiU TSBB09, Lecture A, Klas Nordberg, LiU 14 Particles and energy • Light can also be represented as particles, p photons • The Th energy off a photon h t is i c is the speed of light and depends on the medium c ≤ c0 medium, • c0 = speed of light in vacuum ≈ 3·108 m/s 15 c2 and λ2 must change with the same factor relative to c1 and λ1 c 2 < c1 λ1 ν2 = ν1 λ2 < λ1 • If the speed p of light g changes g from one medium to another, – the frequency ν is constant to make the energy constant – the th wavelength l th λ mustt change h TSBB09, Lecture A, Klas Nordberg, LiU 16 Spectrum Spectrum • In practice, light normally consists of Less number of photons – photons with a range of energies, or – waves with a range of frequencies – This mix of frequencies/wavelengths/energies is called the spectrum of the light More number of photons E E • The spectrum gives the total amount of energy for each frequency/wavelength/energy • Monochromatic light consists of only one frequency/wavelength Same total energies ”Natural light” TSBB09, Lecture A, Klas Nordberg, LiU 17 Classification of light spectrum λ λ – Can be produced by special light sources, e.g., lasers Monochromatic light TSBB09, Lecture A, Klas Nordberg, LiU 18 Polarization • The Th electromagnetic l t ti fifield ld h has a direction di ti – Perpendicular to the direction of motion • The polarization of the light is defined as the direction of the electric field • Natural light is a mix waves with polarization in all possible directions: unpolarized light • Special S i l lilight ht sources or filt filters can produce d polarized light of well-defined polarization 19 TSBB09, Lecture A, Klas Nordberg, LiU 20 Polarization Polarization • Circular/elliptical Ci l / lli ti l polarization l i ti • Plane polarization – The electric field vector rotates – Can C b be constructed t t d as th the sum off two t plane l polarized l i d o waves with 90 phase shift – The electric field varies only y in a single g p plane Electric field TSBB09, Lecture A, Klas Nordberg, LiU 21 Coherence Radiometry • The Th phase h off th the lilight ht waves can either ith b be – random: incoherent light (natural light) – in a systematic relation: coherent light • Light radiation has energy – Each p photon has a p particular energy gy related to its frequency (E = h ν) – The number of photons of a particular frequency gives the amount of energy for this frequency – Described by the spectrum – Unit: U it Joule J l (or ( Watt W tt second) d) – Is usually not measured directly • Coherent light is usually related to monochromatic light sources • Compare a red LED and a red laser – Both produce light within a narrow range – The Th LED lilight h iis iincoherent h – The laser light is coherent TSBB09, Lecture A, Klas Nordberg, LiU • Conversely: plane polarized light can be decomposed as a sum of two circular polarized 22 waves that rotate in opposite directions 23 TSBB09, Lecture A, Klas Nordberg, LiU 24 Radiometry Radiometry • The power of the radiation, i.e., the energy per unit time,, is the radiant flux p – Since the energy depends on the frequency, so does the radiant flux – Unit: Watt or Joule per second – Is I usually ll nott measured d di directly tl TSBB09, Lecture A, Klas Nordberg, LiU • The radiant flux per unit area is the flux density – Since the flux depends on the frequency, so does the flux density – Unit: Watt per square meter As the energy through a specific – Can be measured directly! area during a specific time interval • Irradiance: flux density incident upon a surface y emitted from • Excitance or emittance: flux density a surface 25 Radiometry TSBB09, Lecture A, Klas Nordberg, LiU 26 Basic principle • For point sources, or distant sources of small extent,, the flux densityy can also be measured per unit solid angle • Based on preservation of energy – A constant light g source must p produce the same amount of energy through a solid angle regardless g of distance to the source • The radiant intensity is constant • The radiant flux density decreases with the square of the distance to the source • The radiant intensity intensit is the radiant flux fl per unit solid angle – Unit: Watt per steradian 27 28 BREAK The radiometric chain Sensor Light source Surface 29 The radiometric chain 30 The radiometric chain Sensor Light source TSBB09, Lecture A, Klas Nordberg, LiU Surface 2 Sensor Light source 1 Surface 2 Light source 2 Surface 1 TSBB09, Lecture A, Klas Nordberg, LiU Surface 1 31 TSBB09, Lecture A, Klas Nordberg, LiU 32 Interaction between light and matter The radiometric chain Sensor Light source 1 • Most types of light-matter interactions can p by y be represented n = the material’s refractive index α = the material’s material s absorption coefficient Surface 2 Light source 2 Medium • Both parameters depend on λ • More complex interactions include polarization effects or non-linear non linear effects Surface 1 TSBB09, Lecture A, Klas Nordberg, LiU 33 Light incident upon a surface TSBB09, Lecture A, Klas Nordberg, LiU 34 Basic principle • When Wh light li ht meets t a surface f Based on preservation of energy: E0 = E1 + E2 + E3 – Some part of it is transmitted through the new media • Possibly P ibl with ith another th speed d and d di direction ti – Some part of it is absorbed by the new media E0 = incoming energy • Usually: the light energy is transformed to heat – Some part of it is reflected E3 = absorbed energy • All these effects are different for different wavelengths! E1 = transmitted energy E2 = reflected energy TSBB09, Lecture A, Klas Nordberg, LiU 35 TSBB09, Lecture A, Klas Nordberg, LiU 36 Refraction Absorption • The light that is transmitted into the new g in medium is refracted due to the change light speed • Absorption implies attenuation of g transmitted or reflected light • Materials get their colors as a result of different amount of absorption for different wavelengths Snell’s law of refraction: α1 sin α1 n1 c2 = = sin α2 n2 c1 – Ex: A green object attenuates wavelengths in the green band less than in other bands. α2 37 Absorption 38 Absorption spectrum • Th The absorption b ti off light li ht iin matter tt depends d d on th the length l th that the light travels through the material • The spectrum of the reflected/transmitted light g is g given by y a = e−αx s2(ν) ( ) = s1(ν) ( ) a(ν) ( ) • a = attenuation of the light (0 ≤ a ≤ 1) • α = the material’s absorption p coefficient • x = length that the light travels in the material TSBB09, Lecture A, Klas Nordberg, LiU TSBB09, Lecture A, Klas Nordberg, LiU s1 = incident spectrum s2 = reflected/transmitted fl t d/t itt d spectrum t a = absorption p spectrum p ((0 ≤ a(ν) ( ) ≤ 1)) 39 TSBB09, Lecture A, Klas Nordberg, LiU 40 Reflection Emission • Highly dependent on the surface type • Independent of its interaction with incident g ((well,, almost…): ) light Light is reflected equally much in all directions independent of α α α α Mirror Lambertian surface – Any object, even one that is not considered a light source source, emits electromagnetic radiation • Primarily in the IR-band, based on its t temperature t • More on this in the lecture on IR sensors A real surface is often a mix between the two cases TSBB09, Lecture A, Klas Nordberg, LiU 41 Scattering TSBB09, Lecture A, Klas Nordberg, LiU 42 Scattering • All mediums (other than vacuum) scatter light – Examples: air, water, glass • We can think of the medium as consisting of small p particles and with some p probability y they y reflect the light – – – – In anyy p possible direction Different probability for different directions Weak effect and roughly g yp proportional p to λ-4 In general, the probability depends also on the distribution of particle sizes TSBB09, Lecture A, Klas Nordberg, LiU Medium 43 TSBB09, Lecture A, Klas Nordberg, LiU 44 Scattering The plenoptic function • Scattering is not an absorption g ray y does not travel • It rather means that the light along a straight line through the medium • At a point x = (x1,x2,x3) in space we can g energy gy that measure how much light travels in the direction n = (n1,n2,n3), knk = 1 – There is a p probability y that a certain p photon exits the medium in another direction than it entered. • Examples: p – The sky is blue because of scattering of the sun light – A strong g laser beam becomes visible in air n x TSBB09, Lecture A, Klas Nordberg, LiU 45 The plenoptic function 46 A light camera • A (light) camera is a device that samples the plenoptic function in a particular way • Different types of cameras sample in different ways • The plenoptic function is the corresponding p g radiance intensity y function – p(x,n) (5-dim since x is 3-dim & n is 2-dim) • Can also be a function of – Pinhole-camera – Orthographic camera – Push-broom camera – Light-field Light field camera –… – Frequency ν – Time t – p(x,n,ν,t) (7-dim) (7 dim) – (Polarization) TSBB09, Lecture A, Klas Nordberg, LiU TSBB09, Lecture A, Klas Nordberg, LiU 47 TSBB09, Lecture A, Klas Nordberg, LiU 48 The pinhole camera The pinhole camera model Each point in the image plane is illuminated by a single i l ray passing i through the aperture • The most common camera model is the pinhole camera p – Swedish: hålkamera The aperture through which all light enters the camera • An ideal model of the camera obscura The image plane This is where we measure the image F an ideal For id l pinhole camera the aperture is a single point The camera ffrontt TSBB09, Lecture A, Klas Nordberg, LiU 49 The pinhole camera model 50 The pinhole camera model The image g p plane and the camera center define a camera-centered coordinate system (x1,x2,x3): • M Mathematically th ti ll we need d only l kknow th the location of the image plane and the aperture – The rest is physics + practical implementation – In fact, it suffices to know the aperture (why?) x1,xx2 are parallel to the image plane, x3 is perpendicular to the plane and defines the viewing direction of the camera Principal or optical axis P=(x1,x2,x3) is a point in 3D space • In the literature literature, the aperture point is also called – camera center – camera focal point TSBB09, Lecture A, Klas Nordberg, LiU Q ( 1,y2) is Q=(y i the projection of P 51 f = focal distance, the distance between the image plane and the 52 camera center The pinhole camera model The pinhole camera model • If we look at the camera coordinate system along the x2 axis: • R is the point where the optical axis g p plane intersects the image – The principal point or the image center • The (x1,xx2) plane is the principal plane or focal plane • The green line is the projection line of point P – All points on the line are projected onto Q – Alternatively: Alt ti l th the projection j ti liline off Q Two similar triangles give: 53 The pinhole camera model ! à y1 x1 = − xf 3 x2 y2 TSBB09, Lecture A, Klas Nordberg, LiU or y1 = − fxx1 3 54 The virtual image plane • The projected image is rotated 180o relative to how we “see” the 3D world • Looking along the x1 axis gives a similar expression p for y2 • This can be summarized as: à −y1 x1 = f x3 – Reflection in both y1 and y2 coordinates = rotation • Must be de-rotated before we can view it – In the film based camera, the image is manually rotated – In the digital camera this is taken care of by reading out the pixels in the “rotated” order ! • Mathematically this is equivalent to placing the image plane in front of the focal point 55 TSBB09, Lecture A, Klas Nordberg, LiU 56 The virtual image plane The virtual image plane • Projection lines works as before: from P g the focal p point and intersect at Q through • This defines the virtual image plane P= A point in 3D space – Cannot C tb be realized li d iin practice ti – Produces the same image as the rotated image from the real image plane Q O= The camera focal point • Easier to draw? TSBB09, Lecture A, Klas Nordberg, LiU 57 BREAK The projection of P onto the virtual image plane à ! à y1 x1 = xf 3 x2 y2 ! 58 Lenses vs vs. infinitesimal aperture • Th The pinhole i h l camera model d ld doesn’t ’t work k iin practice since – If we make the aperture small, too little light enters the camera – If we make the aperture larger, the image becomes blurred • Solution: we replace the aperture with a lens or a system of lenses 59 TSBB09, Lecture A, Klas Nordberg, LiU 60 Thin lenses The object plane • Th The object bj t plane l consist i t off allll points i t th thatt appear sharp when projected through the lens onto the image plane • The object plane is an ideal model of where the “sharp sharp points” points are located • The simplest model of a lens • Focuses all points in an object plane onto the image plane object plane image g plane a b 61 Thin lenses • The thin lens is characterized by a single parameter: the focal length p g fL 1+1 = 1 a b fL • To change a (distance to object plane), we need to change g b since f is constant 63 • a = ∞ for b = fL ! – In practice: the object plane may be non-planar: e.g. described by the surface of a sphere – The shape of the object plane depends on the quality of the lens (system) – For thin lenses the object plane can often be approximated as a plane TSBB09, Lecture A, Klas Nordberg, LiU 62 Diffraction limited systems • Due to the wave nature of light, even when various lens effects are eliminated,, light g from a single 3D point cannot be focused to an arbitrarily small point if it has passed an aperture • For coherent light: – Huygens's yg p principle: p treat the incoming g light g as a set of point light sources – Gives diffraction pattern at the image plane TSBB09, Lecture A, Klas Nordberg, LiU 64 Diffraction limited systems Example: 1D Diffraction limited systems • E Each h point i t along l th the aperture, t att position iti x’, ’ acts as a wave source • In the image plane, at position x, each point source contributes with a wave that has a phase difference Δφ = 2π x’ sinθ / λ relative the position at the centre of the aperture • θ is the angle from point x to the aperture, and assuming that x’ << x it follows that sinθ ≈ x / f • We g get: Δφ φ ≈ 2π x’x / ((λ f)) x’ = vertical position in the aperture TSBB09, Lecture A, Klas Nordberg, LiU 65 Diffraction limited systems 66 Diffraction limited systems • The principle of superposition means that the resulting g wave-function at the image g plane is a sum/integral of the contributions from the different light sources: Resulting wave-function TSBB09, Lecture A, Klas Nordberg, LiU • This phenomena generalizes to 2D: – The resulting g wave-function is the 2D FT of the incoming spatial amplitude (function of x’) • Example: a circular aperture of diameter D Amplitude of incoming light 67 First order Bessel function 68 The Airy disk from a single 3D point The Airy disk • The smallest resolvable distance in the image g p plane,, Δx,, is given g byy ψ’ Distance to first zero point in ψ(x) ψ lens focal length lens diameter camera front focal plane image plane TSBB09, Lecture A, Klas Nordberg, LiU The Airy disk, the image of a circular pattern projected j t d iinto t the image plane light wavelength 69 The Airy disk TSBB09, Lecture A, Klas Nordberg, LiU 70 The point spread function Conclusions: C l i • The image cannot have a better resolution than ∆x • No need to measure the image with higher resolution than Δx ! • The Th Airy Ai disk di k iis also l called ll d point i t spread d ffunction ti – or blur disk, circle of confusion – Modulation M d l ti transfer t f function f ti (MTF) • In general the point spread function can be related to several effects that make the image of a point appear blurred – Diffraction – Lens imperfections – Imperfections in the position of the image plane • Be aware a are of cameras with ith high pi pixel el resolution and high diffraction – Image resolution is not defined by number of pixels in the camera! • Often modeled as constant over the image – Can be variable for poor optical systems TSBB09, Lecture A, Klas Nordberg, LiU 71 TSBB09, Lecture A, Klas Nordberg, LiU 72 Depth of field Depth of field • We W have h now placed l d a llens att th the aperture t – Points that are off the object plane become blurred proportional to the displacement from the object plane L) d ≈ 2 ∆x a(a−f Df • Due to the point spread function, it makes p blur in the order of Δx sense to accept L – This blur will be there anyway due to diffraction • Depth of field d is the displacement along the optical axis from the object plane that gives blur ≤ Δx TSBB09, Lecture A, Klas Nordberg, LiU • F For a camera where h a < ∞, an approximation (assuming d << a) for d is 73 a = distance di t ffrom lens l tto object bj t plane l fL = lens focal length D = lens diameter Δx = required q image g p plane resolution d = depth of field Depth of field 74 The F-number F number • fL/D iis th the F-number F b off the th lens l or llens system t • For a lens where a = ∞, points that are further awayy than dmin are blurred less than Δx where • Example – A typical F number of a camera = 8 – Blue light = 420 nm wavelength – Airy disk diameter Δx = 1.22 λ F ≈ 4 μm fL D dmin i = 4 ∆x • For a lens with fL = 15 mm we get – d ≈ 0.6 m at a = 1.5 m – dmin ≈ 1.8 m at a = ∞ TSBB09, Lecture A, Klas Nordberg, LiU 75 This means that the depth of field is within a manageable bl range TSBB09, Lecture A, Klas Nordberg, LiU 76 Lens distortion Thin lenses and the pinhole camera • b is i th the di distance t ffrom th the llens/aperture / t to t the th image i plane • A lens or a lens system can never map g lines in the 3D scene exactly y to straight straight lines in the image plane • Depending on the lens type type, a square pattern will typically appear like a barrel or a pincushion – This is the focal length of the pinhole camera, had there not been a lens. Same as the pin-hole camera focal length f • a is the distance from the lens/aperture to the object plane • a and b are related by 1/a + 1/b ≈ 1/fL, where fL is the focal length of the lens. Often b is variable • All points within the field of depth will be projected with maximum sharpness ∆x on the image plane • The geometric effect of a lens in the aperture is that – The camera center is placed at the center of the lens – The effective focal distance of the pinhole camera becomes b TSBB09, Lecture A, Klas Nordberg, LiU 77 Lens distortion 78 Radial lens distortion • Thi This effect ff t is i called ll d lens l di distortion t ti (geometric ( t i di distortion) t ti ) and can, in the simplest case, be modeled as a radial distortion Position according Ob Observed d point i t to the pinhole camera model (y1, y2) = correct image g coordinate (y1, y2) = r (cos θ, sin θ) (y’1, y’2) = real image coordinate (y’1, y’2) = h(r) (cos θ, sin θ) Barrel distortion No distortion TSBB09, Lecture A, Klas Nordberg, LiU y2 y1 Pincushion distortion • The observed positions of points in the image are displaced in the radial direction relative the image center as described by the pinhole camera model model. 79 TSBB09, Lecture A, Klas Nordberg, LiU 80 Radial lens distortion Lens distortion • Whi Which h di distortion t ti ffunction ti h is i used d depends on the type of lens and other practical considerations: • h is approximately a linear function with g some non-linear deviation,, e.g. The deviation from f a linear function usually grows with r • Once modeled, we can compensate for the distortion 81 – Number of parameters – Invertibility • More complicated distortion models include angular dependent distortion • Cheap lenses ⇒ significant distortion • Almost no distortion ⇒ expensive lenses TSBB09, Lecture A, Klas Nordberg, LiU Vignetting Vignetting • Even if the light that enters the camera is constant in all directions,, the image g plane p will receive different amount of illumination • Sometimes used as a photographic effect • But is usually unwanted • Can be compensated for in digital cameras 82 • This effect is called vignetting TSBB09, Lecture A, Klas Nordberg, LiU 83 Image from a digital camera with a very light lens 84 Mechanical vignetting B The cos4 law • W We can see the th aperture t as a light li ht source iin th the form of a small area that illuminates the image plane α Light from a larger solid angle point A is focused emitted from p here A Light from a smaller solid angle emitted from point B is focused here TSBB09, Lecture A, Klas Nordberg, LiU 85 The cos4 law 86 Chromatic aberration • This effect exists also in lens-based cameras • This means that, in general, there is an attenuation of the image towards the edges of the image, approximately according to cos4α • Can be compensated for in a digital camera TSBB09, Lecture A, Klas Nordberg, LiU – The flux density decreases with the square of the distance to the light g source: cos2 α – The effective area of the detector relative to the aperture varies as cos α – The effective area of the aperture relative to the detector varies as cos α • The refraction index of matter (lenses) is wavelength g dependent p – Example: a prism can decompose the light into its spectrum 87 – A ray of white light is decomposed into rays of different colors that intersect the image g p plane at different points 88 Chromatic aberration Sometimes clearly visible if you look close to the edges through a pair of glasses 89