Alpha Transparency Breakdown

Everything comes down to optimization. A topic that can be debated, usually uncivilly for hours around a table. There are tons of methods to optimize, geometry, materials, shaders, all of which boil down to one simple word: Complexity.

Often it is important to view complexity as a balancing act. We can offload some complexity in areas by making other things more complicated- as backwards as that sounds. An example of this is Alpha masking. This means we aren’t adjusting the RGB values, but the alpha which is used as a transparency channel.

That’s a really technical summary. We can just call it a cutout mask.

This has an added benefit of creating an opportunity of variation. We can reuse an asset over and over again replacing a simple texture and get many more. In some projects there are many more layers you can use with alpha masking and textures to offload complexity. A lot of game studios will customize shaders, textures, and assets to all work cohesively. This is to allow for an optimized real-time experience in the renderer’s engine.
We use cut out masks in cases like decals, or to avoid adding more geometry then we need to for the context of our content. For example, we have jack-o-lanterns. In the past we have distributed geometric based Jack-O-Lanterns. We cut out the shape just as you would in real life, this left real geometry that light (technically the shaders) in the renderer can interact with.

As stated above, sometimes the context of an object doesn’t need that detail. In the case of these Jack-O-Lanterns, we are going to say we don’t care about the depth of that cutout. So we can offload that complexity to a mask.

This has an added benefit of creating an opportunity of variation. We can reuse an asset over and over again replacing a simple texture and get many more. In some projects there are many more layers you can use with alpha masking and textures to offload complexity. A lot of game studios will customize shaders, textures, and assets to all work cohesively. This is to allow for an optimized real-time experience in the renderer’s engine. 

Let’s slowly get technical again. What is a cutout mask and how does it work? A cut out mask is just an image that tells the rendered what to make transparent and what to make opaque.

Black gets interpreted to transparent and white gets interpreted to opaque. In our modeling programs we can use a geometry’s UV map to create an opportunity to simplify complexity through alpha masking.

In blender, we took our pumpkin’s UV map and created a separate material channel to handle the cutout.

When we do this correctly and map the textures to their material attributes we get a neat result:

When we do this correctly and map the textures to their material attributes we get a neat result:

For Enscape we can simply go to the custom asset editor and import our pumpkin. Select the material we have set up to hold our cutout mask. We can scroll down and under transparency make sure type is set to “Cutout”. Then click on the folder icon that will take us to an explorer to choose a mask.

(Note: You should only use purely black and white image files)

When we do this, it changes the pumpkin to have a cutout face.

You can make any personal adjustments, then take your Enscape Asset Library thumbnail image, and save your asset.

Here is a scene with a lot of alpha masked layers. Heck, even Enscape uses alpha masking on their grass. The grass, the hay, and the pumpkins all have cutouts to help them work with the scene better as well as get more out of some assets without being destructive.

EXTRA:
For educational reasons we wanted to keep this tutorial straightforward and simple. There is a ton of area we waste on the image file that could be simplified, but for distribution and customization purposes we opted not to. If this is your first experience editing a cutout or diving into materials, feel free to mess around. There is no harm in trying things out!

Please take full advantage of improving this workflow and see what you get. Let's quickly look at this pumpkin and see what issues we’d run into if we carved out the face’s geometry. In this example I quickly ran a boolean on the Pumpkin to cut the face. It results in a lot of shading issues and incredibly overly complex geometry. In most cases, this is just wasted data on a pumpkin that could be used to provide more complexity in different areas of your scene.

You can create your own masks in any image editor. We can make our own mask in MS Paint. We’ve included a template to make that easier

Open MS Paint, open the template image. Anything that is in the white rectangle will be cutout. Everywhere that is black on the template is outside the UV Map so it isn’t visible on this cutout. 

We have also included a UV Map template

If you’re more familiar with image editing software, you can import this as a layer in your editing program and then use it to guide your mask’s boundaries. If you trade the UV grid like a boundary you can understand it better. Everything outside the boundary (Red) will not be seen, anything inside the boundary (Green)  will be seen.

You can also take a stab at using AI to get some procedural masks. We used Stable Diffusion in blender with the prompt saying “Black and white mask of a face outline” which is the riskiest bare minimum for a prompt.  As always, be prepared to fix some of the AI’s interpretations. Sometimes that means just adjusting where it is. Sometimes that means adjusting the levels so the blacks and whites are harsher.

You can quickly see how creating one geometry smarter can be leveraged to get more use out of that one object with variations.

If we can help your team with Enscape or Revit please send a message to hello@readthomas.com!