Masking Triangle: A System To Making Sense of Layer Masking

Layer masking is one of the most important fundamentals in digital image editing. It’s THE gold standard technique used by almost everyone in post-processing workflow.

Whether you’re new to image editing or on your way to conquer layer masking, you have to not just know it but master it if you want to produce stunning images like the ones you see on sites such as 500px.

Like others, I struggled to grasp the idea of layer masks at first and wasted a lot of time messing around in Photoshop.

I got there eventually...but it wasn’t until recently that it all “clicked”!

I have never seen it this way: Selections, layer masks and channels are all inter-linked, like a triangle. The first thing that came to my mind was "masking triangle". So I thought I'd put everything together for those that might find it helpful.

If you're new to photography or struggling to figure out layer masking, this is for you.

Even though the methods in this tutorial are explained using Adobe Photoshop, the principles can be applied in most editing software.

1. What Is Making Triangle?

2. Why Masking Triangle?

Remember the time when you first held a camera and you have no clue how to even take a decent image?

You started with the most basic of all, the exposure triangle. It’s a system that associates the aperture, shutter speed and ISO together to help us make sense of complex subjects. 

2.1 The Brain Likes Systems 

2.2 "I Don't Need To Know Masking"

3. Selections

Selections can be either user-defined or pre-defined based on the software’s algorithm. It’s often associated with “marching ants”, a term to describe the moving dotted lines outlining the selection.

A selection is a way to tell the software (almost all software are the same for this) that the desired area is selected. An adjustment can then be applied to this selected area only without affecting other parts of the image that is not selected.

You'll understand more about user-defined vs pre-defined selections in a minute.

3.1 Hard-Edge vs Feathered-Edge

The border of a selection can be a hard-edge (a.k.a. hard selection) or a feathered-edge. 

A hard-edge separates the selection from the image in a clear-cut fashion. When an adjustment is applied, the effect is obvious and stands out from the image when comparing the before and after image.

When an adjustment is applied to a selection with feathered-edge, the effect transitions smoothly into the rest of the image. In this case, the effect doesn’t stand out but look more subtle. 

The two images above are identical. I made a rectangular selection across the middle and applied +30 saturation. The selection in the left image has no feathering (hard-edge) whereas the selection in the right image has 100 pixels feathering.

Without knowing there is a saturation adjustment, you can barely notice it on the image to the right.

3.2 Methods of Creating A Selection

A selection can be user-defined (made by you) or pre-defined by the software (pre-made for you). Generally speaking, it can be made with the following methods:

The details of each is a topic on its own and therefore won’t be covered in this article. If you want to learn more, you can sign-up for the free luminosity masks crash course where this will be explained in more detail in training videos.

3.3 Convert A Selection To Layer Mask and Channel

When you have an active selection, you can apply a layer mask to it. By default, this will conceal the area outside the selection. This means the part of the image that is not selected will be hidden by the layer mask and becomes invisible to you.

If you want to save the selection for later use, go to the Channels panel and click “Save selection as channel”. This saves you time from creating the same selection again, particularly when you have spent a considerable amount of time in a complex selection.

Not to worry if you don't feel you quite get this. All three elements in the masking triangle are inter-related. I promise it will make more sense as you continue to read.

4. Layer Masks

Layer masks have only one purpose - to conceal and reveal part or whole of an image.

Imagine you have a piece of transparency placed on an overhead projector. You take a piece of black cardboard and cut a hole out of the middle. You place it on top of the transparency and now only the middle where you cut a hole out is visible.

Now imagine: The transparency is a layer. The black cardboard is a layer mask. The hole you cut out is a selection.

You can swap the black cardboard into any shapes and sizes. It can cover part or the entire transparency. This goes the same for the hole you cut. It can be any sizes and shapes.

A layer mask can only affect the same layer it is applied to. A dark layer mask conceals what is underneath it but a white layer mask reveals.

In the image above, there is a layer mask (1) applied to the image (2) in Layer 1. The layer mask (1) cannot affect the layer above (3) or below it (4).

4.1 Creating A Layer Mask

It’s pretty simple. 

Select a layer and click on the icon at the bottom of the Layers panel that says “Add layer mask”.  You need to place the cursor on the icon for the description to appear.

This will add a white layer mask to the layer. If you hold down Opt (Mac) or Alt (PC) then click the icon, a black layer mask will be added instead.

By default, all pixel layers do not have a layer mask. If you want one, you have to add it manually like what I’ve explained above.

Adjustment layers come with a pre-added white layer mask. You can paint black directly on to it or create a selection first using one of the methods described earlier, then apply on to the layer mask.

4.2 Convert A Layer Mask To Selection and Channel

5. Channels

This one is often forgotten or “ignored” particularly by beginner photographers. 

Channels panel is where your saved selections are. You can also customize it here to make it however simple or complex you want.

In Photoshop, when you open an image and go to the Channels panel, you will see three grayscale images below the color (RGB) image. These are the channels for red, green and blue color - what I called pre-defined selections.

5.1 Creating A New Channel

First, make a selection.

Go to the Channels panel and click “Save selection as channel”. In Photoshop, a new channel will appear just below the Blue channel.

It's always named “Alpha 1” by default where “1” is the number to keep count on the new channel. Subsequent new channels will be named 2, 3, 4...you get the idea. 

Just like layers, you can click on the words to change the description to something more meaningful to help you remember what is it for. Trust me, things can get real messy when the number of channels stack up.

5.2 Convert A Channel To Selection and Layer Mask

Channels panel comes in very handy when your editing work involves multiple selections. It really saves you time. Even if you need to create a new selection, you can modify the saved ones to help you make the new selection faster. 

To convert a channel to a selection, hold down Cmd (Mac) or Ctrl (PC) and left click on a thumbnail image in Channels panel. This reactivates the selection and you might see the marching ants again.

Don't panic if you don't because Photoshop only displays marching ants around pixels that have at least 50% opacity.

When the amount of feathering for the edges are too high, a message often pops up that says "Warning: No pixels are more than 50% selected...". This just means your selection is active but you won't visualize it with the marching ants.

Just click OK, ignore it and continue. (one of the many warnings you can ignore in life!)

Once you have the selection reactivated, apply it as a layer mask with the methods explained above.

6. The Missing Guide

Each element of the masking triangle are inter-related. Changes to one will change the other two.

The main purpose of the masking triangle is not to confuse you even more. It serves as a system (or a guide, whichever you prefer to call) to help you make sense of the elements in layer masking.

Interesting or useful (either, hopefully)?

Let me know in the comment below.

For more tutorials on image editing technique, please check out the editing technique resource page!