Using an Image as its Own Photoshop Layer Mask
The usual way to create a layer mask in Photoshop is to paint with black on a white background, or to paint with white on black, following the contours of the image and the objects being masked. Since this is the case, why not create the mask directly from the image itself and avoid having to repaint what you already have?
Think about it for a minute. What exactly is a layer mask anyway? As a black and white channel added to your image, it can contain values from zero to 255 at each pixel just like the standard red, green and blue image channels (or the equivalent in 16-bits per pixel mode). The only difference between a layer mask and a regular channel is what it does, not what it is. Image channels can be manipulated in pretty much the same way regardless of type. You can paint on your mask with a brush just as you can on the image itself. You can adjust levels, apply filters, copy and paste on a mask, and so on, just like you are accustomed to. But rather than contributing a component to the image's color, it is usually used to selectively hide or reveal portions of the image's data, affecting the way the associated layer blends in with the ones underneath.
The original source for the red, green and blue channels of a photograph is obviously the data recorded by your camera. You could paint them from scratch instead of course, but what you would end up then would be a digital painting, not a photograph. So why is it that the typical layer mask starts with a blank canvas, relying mainly on a black and white paintbrush to add the needed details? Why shouldn't a mask for an image start with the image data from your camera just like the rest of your image does?
This week I want to describe a couple techniques that take advantage of this idea, but you can think of these really just as a starting place, a proof of concept perhaps for something you may never have thought of. Once you catch on to the idea, you will likely find other, similar techniques to improve your own images. It's the idea that's important. Once you see how it works, you can take that idea and run with it.
Tinting and Duotoning
A duotone image is created by applying one color to the highlights of an image and contrasting one (often simply black) to the shadows. As with most things, there are plenty of ways to create tinted or duotoned images in Photoshop, but here's one that demonstrates what I mean by using an image as the source of its own layer mask.
Layer mask for duotone highlights
The first thing you need to is to create a new channel containing a black and white version of your image. To do so, select your entire image with Control-A (Command-A on Mac OS X). Then copy it to the clipboard. If your image consists of a single layer, you can do this with a simple press of Control-C for "copy" (again, make that Command-C on OS X). If you have multiple layers, you can use Edit >> Copy Merged from the menu, which turns out to be Shift-Control-C (or Shift-Command-C) if you're really into the keyboard shortcut thing. Now go to the Channels panel and click on the "Create new channel" icon at the bottom margin, underneath the list of current channels. Now simply use Edit >> Paste (or Control-V) to paste your image into the new channel. Since a channel can't contain multiple colors, Photoshop will automatically convert your paste to black and white.
Now use Select >> Load Selection, choosing your new channel as the source, to convert your black and white rendition into a selection. Any point that was pure white or pure black will now be completely selected or completely deselected, but most points in your image will be somewhere in between. In other words, you now have your image highlights selected, more or less. Each pixel will be partially selected as determined by how bright it is in your source channel.
With that selection having been made, you can now create a new Photo Filter adjustment layer and choose whatever color and density you wish to tint the highlights of your image. Your selection will be converted to a layer mask on this new adjustment to constrain the effects of the filter to the areas you had selected.
At this point, you will still have many shadow areas at least partly selected since only completely black pixels would be absent completely from the resulting selection mask. To fine tune things, open Image >> Adjustments >> Levels, grab the black point slider and raise it to exclude more of your image from being affected. If you want to look at the mask itself rather than your actual image while doing this adjustment, At-click (Option-click on Mac OS X) on the layer mask icon in the Layers panel before opening Levels. You can return to the regular image view by clicking on anything else in the Layers panel when done.
If you want to apply a tint to the shadow areas as well, follow these exact same steps but use Select >> Inverse before creating the Photo Filter layer. This way, your shadows will be white and your highlights black in the mask, and your Photo Filter will affect the shadows instead of the highlights. You can create one adjustment layer for the highlights and a different colored one for the shadows to affect both in a different way.
As we saw in the previous technique, a channel can be converted into a selection, and a selection can be converted into a layer mask. All three are basically just different ways of representing the others. With this in mind, we can also use an image as a selection when applying a filter to that image, in this case a sharpening filter.
Channel selection for edge sharpening
First, create a new channel containing a black and white version of your image as we did in the previous example. This will get us a mask that represents bright areas as white, dark areas as black, and everything else in between.
But what we need this time is a mask that selects the areas of transition between dark and light, without also selecting either end of the tonal scale. Thankfully, Photoshop has a very useful feature under Filter >> Stylize >> Find Edges. This filter has no settings so don't be surprised that it doesn't present you with a dialog box. When you use it, you'll end up with a channel image that looks like a line drawing version of your image. Edges will appear as black lines and everything else will be white. Or close to it. You can improve up the results by using Levels to increase the black point and decrease the white point to make the line edges cleaner. As a starting point, bump up the black point to nearly where the gray (midpoint) slider is, and then lower the white point until you like what you see.
Now you'll need to apply a small degree of Gaussian blur so the line edges aren't quite so sharp. Somewhere between one and three pixels should suffice, depending on your image dimensions.
You're now ready to convert this channel to a selection. Go to Select >> Load Selection as we did in the previous example. Pick the channel you just created, and click on "OK." If you look at the "marching ants" that surround the resulting selection boundaries, you'll notice that the majority of your image is now selected, with the edges excluded. Check the very sides of the image to see what I mean. If you have marching ants along the sides, top and bottom of your image, you have most of the image selected. This makes sense since the Find Edges filter left us with black lines on a white background. To reverse things, use Select >> Inverse now and you should end up with a selection that consists just of the edges in your image, not the "non edges."
Once you are satisfied that you have the right selection created, you will probably find that the marching ants boundaries are a tad bit distracting. To make them go away without affecting your actual selection, turn off the checkmark next to View >> Show >> Selection Edges. The marching ants will disappear which generally means you no longer have anything selected, but in this case you still do. Your original edge selection is still present, but the indicator dashed lines of the marching ants are simply no longer being displayed. When finished, click on the RGB channel so you can see your actual image again rather than the edge selection channel.
You're now free to use your favorite sharpening tool with the knowledge that its effect will be constrained to your edge selection. Suppose we use Filter >> Sharpen >> Smart Sharpen. You should find that you can push the amount slider up much further than you're used to without nasty artifacts resulting. Since you only have the edges selected, areas that are supposed to not be edges won't get sharpened no matter how aggressive you get.
There are plenty of other ways you can use an image as its own layer mask. But hopefully this will get you started thinking about the possibilities.