This tutorial will demonstrate how to extract a clean mask from a photo of a person allowing for hair, using channels and levels. The general concepts may be intuitively applied to other applications.
First, if you want to try this yourself, use that attached image, and open it in Adobe Photoshop.
1. Balancing the channels
Open the Channels palette to inspect the various color channels.
Color channels tell how much of each color is in the image. An RGB image like this photo will contain 3 seperate color channels, one indicating the amount of Red, one for Green and one for Blue. White in a channel means that the color value is full (255), and black is empty (0).
Look for the channel with the highest contrast edge outlining the hair. The hair should be a flat color of very low contrast within the edge, and the backdrop should be black or quite dark, also with low contrast. This way it will be easy to define the head with an Alpha channel.
It is easy to see however, we don't have a perfectly black and white image to use as a mask yet, so we need to make the background darker and person lighter.
a) Make a copy of the Red channel by dragging it onto the New Channel button, and rename it Person Alpha.
b) Then make a copy of the Green Channel and make sure it's selected.
c) Adjust levels on the Green Channel:
Image > Adjustments > Levels... (Ctrl L)
Now the idea here is to isolate the person from the background. On other images you may need to also adjust the other channel in the same way so the subject is isolated, but that is not required here.
d) Drag the Green copy channel onto the Load channel as selection button.
e) Click to the Person Alpha channel, and then press Ctrl H to hide the selection so we can see what's happening.
f) Now significantly lighten the person region:
Image > Adjustments > Levels... (Ctrl L)
When compositing the person into another scene, the background will want to be totally transparent, and the person solid. So bump the shadows to black, and the person to pure white. To do this:
g) Press Ctrl+D to clear the previous selection.
h) Adjust levels on the Person Alpha channel:
Image > Adjustments > Levels... (Ctrl+L)
Now slowly drag the black input level upwards until the whole background turns black, and slowly drag the white input level downwards until most of the person appear White.
It may take a number of tries to find the perfect balance.
Be mindful of what is happening throughout the image as the levels are adjusted. If the white levels get too low, some hair will disappear into the background. And if the black levels get too high, the background will glow through the edges of the person.
2. Applying the mask
a) Drag the Person Alpha channel onto the Load channel as selection button.
b) Go to the Layers palette, and make a copy of the Background layer, by dragging it onto the New layer button.
Double click on the new layer Background copy, and rename it Person.
c) Click Add layer mask on the Layers palette to add the selection as a mask to the Person layer.
d) Hide or Delete the original Background layer.
Click the little black and white thumbnail on the Person layer to paint on the layer mask.
Clean up the person mask by filling unwanted gaps in the hair and jumper. Use the Brush tool to fill in holes in the person by painting white on the holes, or black where you require transparency.
3. Tweaking the edges
a) Make a new layer, drag it beneath the Person layer, then paste a scene from the project it will composited into, or else fill the background with an average color of the scene, or make a gradient behind the person layer, to test the mask over a range of colors.
Note: I resize the person layer to show more of the scene in Layer1. Select the Person layer, select the crop tool, drag the crop larger then Ok.
If the person is going to be positioned over a background, the edges could be quite dark as some of the background is coming through, so to fix this:
a) Hold down Ctrl and click on the Person layer mask thumbnail to select the mask's transparency.
b) Contract the selection by 1 pixel.
Select > Modify > Contract... 1 pixel
c) Invert the selection. Ctrl+Shift+I
d) Create a Curves adjustment layer, by clicking on the Layers palette, and selecting Curves...
Drag the lower part of the Curve to the left and another to help bring the line back to linear.
e) Holding down Alt, move the mouse cursor between the Curves and Person layers, until the cursor changes to two overlapping circles, and then click.
Doing this creates a clipping group, so that the Curves adjustment will only effect the Person layer.
In a clipping group, the bottom layer acts as a mask for the upper layers.
Curving the edges so may create a noticable white line around the hard edges of the person. This may be softened by Gaussian Blurring the Curves layer mask by one or two pixels.
Though blurring the Person layer mask may produce undesirably trippy results.
If your happy with the image, remove the scene (Layer1) crop the image to size, and export: File > Save for Web... (Alt Shift Ctrl S) Select PNG-24 and enable transparency.
That about does it. Play around with the settings of each step to refine this image, however, these settings and exact steps will be different for each image.
My biggest recommendation for this technique is try and plan the shot so that it's easy to select the subject in the mask. I normally try and use a red, green, or blue background, to make it easier to isolate the subject, making sure the subject has as little as possible of the colour on them. Otherwise using a black or white background also helps.