Gamut Warnings and What to Do About Them
Printers can only display a limited range of colors, known as their gamut. Photoshop can provide warnings for image colors that lie outside your printer's gamut via soft proofing. But the real question is, what do you do about them once you identify them?
Fixing out of gamut colors can be broken down into a two-step problem: finding the problem colors, and fixing them. At least some readers will already be familiar with this first step, but this includes you, bear with a minute while I get everyone on the same page with a brief introduction to soft proofing.
Image files, regardless of source, exist in some color space. This means their colors are defined using some variation of standard color model. For photographic images, I'm going to assume throughout that we are talking about the RGB color model that uses sets of three colors, one each for red, green and blue, to define colors. But not all RGB color spaces are created equal, with some capable of describing a broader range of colors than others. In order to maximize the benefit from the available bit depth, the numbering scheme used by each color space optimizes the range of describable colors to best match the intended purpose. In other words, a monitor profile is optimized to describe colors possible on a given monitor, a printer profile is geared to describing colors unique to specific printer and paper combinations, and so on. Document working space profiles such as Adobe RGB and ProPhoto RGB have been defined by various industry bodies for saving images so that they aren't tied to any specific monitor or printer.
When you print an image, it has to get converted from its current document profile to the profile needed for printer model and the paper you are printing on. This process can either be done automatically for you by your printer, or you can use Photoshop to do it. As you might guess, doing this in Photoshop gives you much more control over the details of how it is done and is to be preferred if you are interested in getting the best results you can. Regardless, converting from one profile to another is generally a process involving compromise. It is rare for the target profile to be capable of describing all source colors correctly. When this can't be done, colors that don't fit are said to be out of gamut.
Photoshop soft proofing provides the tools needed to identify out of gamut colors. To use it, go to the View >> Proof Setup >> Custom menu and select your target printer profile. Check the box for "Black Point Compensation," but make sure that "Preserve RGB Numbers" is not checked. The Display Options at the bottom of the window can be set at your discretion. Some profiles provide data to better handle simulation of paper color than others. Rendering Intent is a complicated subject, but in general you only want to consider "Perceptual" or "Relative Colorimetric." The former will compress the overall gamut of an image evenly so that out of gamut colors will blend in better with their surroundings, while the latter maps out of gamut colors to their nearest in-gamut color while leaving colors that do fit more or less alone. Both alternatives have their pros and cons that you can read more about here. The bottom line though is that to get the best results possible, I'm going to talk you through manually dealing with out-of-gamut colors to make this choice mostly irrelevant. Once all the out-of-gamut colors are gone, the difference between various rendering intents is negligible. Click on the "OK" button when got everything set as you want it.
With soft proofing setup the image you see on your screen will be modified to match as close as possible to how it will print with those settings. To see the out-of-gamut colors that required modification, turn on View >> Gamut Warnings. Colors in your image that lie outside the selected target profile will be overlaid with the Photoshop gamut warning color. What color that is depends on your choice in Edit >> Preferences >> Transparency and Gamut. The default gamut warning color is a neutral medium toned gray which is frankly one of the worst choices for gamut warnings since it's so boring. You want a color that is going to jump out at you so you notice it. I generally go with a brilliant cyan, but depending on your particular image some other color may be better. Pick a color not found in your image if you can.
Now that you've found your out-of-gamut colors, the task at hand is to make all those ugly gamut warnings to go away. If you simply go ahead and print, Photoshop will compensate for your gamut problems automatically based on your selected rendering intent. But again, by taking control yourself, you can often do better. Automatic saves time, but it's not just about time now is it?
You might hear people recommend using Select >> Color Range to select "Out of Gamut" colors as the basis for creating a layer mask for modifying the problem colors. But this is a poor choice since it utilizes your working CMYK profile as the basis, not the profile you just got done selecting for soft proofing. This means it will almost certainly select way more colors than the soft proofing gamut warning color indicates. It might also fail to select colors that should be included depending on the gamut of your printer profile. It would be nice if Adobe would change this so it works, but for this particular purpose, it doesn't. Frankly, even if this method of selection did work, it would be rare for all out-of-gamut colors to be best handled in the same way, making any single selection mask impractical. Instead, it's best to work through the out-of-gamut areas manually, dealing with each problem color one by one.
Gamut problems happen at the extremes, either exceptionally saturated, or occasionally exceptionally bright shades of particular colors. In the image here of mountain wildflowers the magenta petals are clearly a problem. We can either slightly desaturate them, or slightly darken them. But the trick is to decide which.
Use the eyedropper to set your foreground color to a problem color by clicking on it in your image. Now click on the foreground color square in the toolbar to open the color picker. Just as you can soft proof your image to see gamut problems, you can also turn on gamut warnings in the color picker. This seems to be a poorly documented feature, but if you look at the main program menus you'll find that View >> Gamut Warnings has a keyboard shortcut of Shift-Control-Y (or Shift-Command-Y on OS X). The color picker itself doesn't have a menu, but thankfully the same keyboard shortcut works here. Once you open the color picker, hit Shift-Control-Y and the color field will be overlaid with the same gamut warning color you chose in Preferences. Hitting Shift-Control-Y again will turn the color picker gamut warning overlay off again, but you'll want it on for the task at hand.
If you correctly set the foreground color to an out-of-gamut image color, you should now find that the selected color in the color picker will also show out-of-gamut. But you can now tell how close that color is to an in gamut color, just be looking to where the closest non-warning color is. This will let you decide how best to proceed. If the closest acceptable in gamut color is below the problem color, you'll want primarily darken that color in your image. If the best in gamut choice is to the left of the problem color you'll need to desaturate to get there. Or you may decide you need to do a little of each. Regardless, you now have the information to decide for yourself what would look best.
To actually fix things, you could simply use the Dodge and Burn tools to adjust brightness, or the Sponge tool selectively desaturate, but these tools need actual image pixels to work on. Instead, I'd suggest doing this non-destructively by adding two new image layers, each with a specific blending mode. First, hold down the Alt key (Option key on OS X) and click on the "new layer" icon at the bottom of the layers panel. In the resulting pop-up dialog, select a blending mode of "Soft Light" from the Mode dropdown. Click on the "Fill with Soft-Light neutral color (50% gray)" box that appears at the bottom of the window and then click on "OK." Then create another new layer with a blending mode of "Saturation." There is no neutral color for saturation so just leave this layer empty for now.
With these two adjustment layers created, you can paint on one to darken, or paint on the other to destasturate. Set your brush to a low opacity black and paint as needed on the out-of-gamut color areas. Painting on the Soft Light layer will darken colors. If you change your mind, paint over that area again with a white brush. Anything left neutral 50% gray won't modify your image at all. Anything darker than this will darken the underlying image color. Anything brighter than 50% gray will lighten the image. Similarly, painting with any neutral color on the Saturation blending mode layer will destaturate the image at that point proportionately to the brush opacity since neutral hues are by definition unsaturated. If you change your mind, switching to a white brush won't help. Since it too is completely desaturated, painting with white will also desaturate your image. To restore some of the original saturation, use an eraser brush on your Saturation layer instead. By painting appropriately on these two adjustment layers you can have complete control over the brightness and saturation of each point without worrying about modifying your actual image at all.
It often isn't necessary to manually deal with all the out-of-gamut colors since your selected rendering intent will deal with whatever is left. But by manually taking control you can be sure critical colors come out the way you want.