Lytro: Very Cool, and not Like Any Camera You've Ever Used Before
To say the least, when Silicon Valley startup Lytro announced their new light field camera this past June that allows you to refocus images after the fact, I was intrigued. Now that they are available for preorder, I'm even more so. But just how does the darned thing actually work?
Traditional photography captures whatever image falls on its recording medium. The image is a recording of a flat, two-dimensional surface. If the subject is in focus, the resulting image will be in focus. If not, the image will be blurred. While changing the size of the aperture opening can increase or decrease the apparent depth of field, the lens is truly in focus at one distance at a time. Changing focus distance consists of moving the lens closer to or further away from the sensor. If you goof when focusing the lens there's little you can do later. The image has already been recorded. It is what it is.
By contrast, the new Lytro camera records more than just the light falling on the sensor. It records information about the entire "light field" of light rays travelling through the camera body. With this it can extrapolate what the image would look like focused to different distances after the fact. To work this magic, Lytro employs a number of revolutionary ideas.
Mentally picture the space inside the camera through which the light travels. The light refracts through the lens and is focused toward the sensor. To record the path of any given light ray through this space you might think it necessary to know its position at every possible point along its travels. To do so for any meaningful number of light rays would require massive amounts of data. But all you really need is its position at the plane of the lens aperture and the plane of the sensor. Only a single line passes between any two points in space, so once we've captured the data for two points we have sufficient information to calculate the position of any other point along this line. This greatly simplifies the problem of recording information about the entire light field inside the camera.
But it's not enough to build a camera for at least two reasons. First, with countless light rays making up this light field, there would be no practical way of associating any given light ray's intersection point with the plane of the lens aperture with the corresponding intersection point at the sensor plane. You would effectively have two sets of data, one from recorded at the lens plane and one recorded at the sensor plane, but there would be no way to correlate those sets of data. In theory this could work for a single light ray given that you would have only a single point recorded in each set of data. Wherever you found it in one set must match up with where you found it in the other. But with enough light rays to actually define an image, there would be countless "its" in each set and no way to sort out which "it" over here matched up with which "it" over there. And if this weren't enough to shoot down our mentally pictured light field camera, there's another problem to overcome. Any actual camera would have to be more than just mental, it would have to be physical. We'd need two detectors. The detector at the sensor plan is easy enough since the digital sensor already commonly in use in modern cameras is itself such a detector. But there's simply no detector that would be able to digitally record information about the light passing through the plane of the lens aperture without significantly blocking or distorting that light on its travels to the sensor plane.
Instead, the Lytro camera makes use of one of those revolutionary ideas I mentioned earlier. Rather than recording information about the light at the lens plane, it records information it can use to mathematically derive that information. Before describing how they do it, it's worth pointing out that current digital cameras already heavily rely on mathematical algorithms to derive information rather than directly recording it. While each pixel in a digital camera image has red, green and blue components, only one of those three colors is actually recorded by the camera. The other two are interpolated based on data for those colors recorded by neighboring pixels. The precedence of calculating data needed to construct an image is already well established in digital photography. Lytro just does it in a new way.
What Lytro does is to affix an array of tiny micro-lenses immediately in front of a more traditional digital sensor. When you shoot with it, the main camera lens sees only a portion of the entire world in front of you based on its focal length and angle of view. In much the same way, with their size measured in microns, these micro-lenses see only a portion of the image being projected by the camera lens. Each micro-lens projects a version of what it sees onto the camera sensor. Based on where each exists in the array, what you end up with is an overlapping grid of varyingly off-axis snippets of the overall image. If the camera's main lens is analogous to a human eye, you can think of this micro-lens array as being somewhat like the compound eye of a fly or other insect.
The image projected by one micro-lens relates to those projected by its neighbors in a mathematically predictable fashion. Through a series of complicated formula, Lytro is able calculate the paths of the light rays that must of created those snippets. Thus, without an actual sensor at the plane of the lens aperture, Lytro can figure out the missing information. With this, it has everything it needs to determine what the image would look like focused to varying distances.
All this obviously has an impact on image resolution. The number of micro-lenses in the array clearly will impact resolution, but so too will the role they serve. With each micro-lens creating multiple overlapping snippets of image on the camera sensor, it takes more sensor pixels to create a final image pixel than would be the case with a more traditional digital camera. Newer digital cameras boast ever increasing numbers of megapixels, but few of us yet feel we have any to spare. In order to pack enough pixels in such a small size, the pixels in current compact cameras is under two microns though so if one were to construct a full frame 24x36mm sensor with that same density you would end up with well over 200 raw megapixels. The reason why we haven't yet seen an actual camera with such a pixel pitch is in part due to the diminishing returns such a design would yield. Sharpness loss due to diffraction is already beginning to impact true resolution at current pixel densities. And diffraction gets progressively worse at smaller apertures. But since Lytro can focus after the fact, there's no reason to stop the lens down before shooting to get sufficient depth of field. The lens in the new Lytro camera is a fixed f/2 lens, something unthinkable in a more traditional digital camera.
Lytro isn't really saying what the resolution of their images will be. The quoted camera resolution on their website is a cryptic eleven mega-rays. I'm presuming this means it can determine the paths of eleven million light rays through the light field but it isn't clear whether that refers to the number of micro-lenses, the number of pixels in the sensor that records the raw data, or something else entirely. The company has promised resolution exceeding 1080p which is obviously enough for many uses, but all that can be seen so far is the samples in the image gallery on their website which are clearly less than this. The documentation on their site states that the full image resolution is accessible using the desktop software they provide with the camera.
The physical design of the new Lytro camera is also rather revolutionary. Measuring 1.6 x 1.6 x 4.4 inches with a lens at one end and an LCD screen at the opposite end, it looks nothing like traditional camera. It's essentially a featureless aluminum block with none of the customary camera controls visible. There a rubberized grip covering one end of the block, a subdued on/off button and shutter release button, a slider for the 8x optical zoom, a USB port for image upload and for charging, and little else evident. You set exposure by tapping on the touch screen. The Li-Ion battery is internal and not user replaceable. The storage memory is also built in. Lytro produces the camera in models with either 8 GB or 16 GB internal memory.
Let me be clear. I don't have a Lytro camera yet. This is not a hands on review. But I have spent some time studying the information that is available from Lytro. The mastermind behind Lytro is Ren Ng who did his doctoral dissertation on "Digital Light Field Photography" at Stanford University in 2006. For those interested in the math behind the magic you can read his entire 200-page paper on the Lytro site.
If you haven't yet heard of Lytro, you will. If you haven't at least visited their site to see their sample images, you should. These are not images just to be looked at; they are images to be played with. Click on them with your mouse and they refocus to that point. Click elsewhere and the focus changes again. Lytro calls them "Living Pictures". I call them very cool indeed.
Update 10/30/2011 - Reader BL emailed me that "Concerning Lytro's camera, the plenoptic photography concept was invented in 1903 by Frederic Yves, and refined with micro-lenses in 1908 by Gabriel Lipmann (he won the Nobel Prize for physics)." I hadn't meant to imply that Lytro had invented the concept, only that they were now popularizing and commercializing it. As BL goes on to point out, Plenoptic cameras have already been available for sale since 2007 by the German company Raytrix. An interesting data point is that the Raytrix R11 has an 10.7 megapixel CCD sensor and yields 3MP images. Still no definitive word on the final resolution of the Lytro images.