Ambient Occlusion with Transparency!

I ran into a scene where I wanted to render an ambient occlusion (AO) pass, but certain objects were using transparency maps as cutouts. So when I rendered the AO pass, those objects with transparency maps turned out solid...not the desired result.

Disclaimer: This is some-what of a half solution for AO with transparencies. AO works based on geometry proximity and does not take into account mapping. So this solution is an approximation. With that said, lets take a look at AO with transparencies.

I'm going to explain how you can render an AO pass while keeping the transparency of certain objects, just using the mental ray and A&D shaders. The example I'm using here is a tree textured with A&D materials.

The leaves on the tree are simply planes, but when rendered you see the leaves, because the planes have a alpha-map of the leaves in the cutout slot of the A&D material.

When you render an AO pass (click here to go to my AO in mental ray posting), the transparency on the leaves go away and the occlusion shows the planes. Of course this is not what we're wanting.

So to render an AO pass with transparency, there are regular objects and there are the objects that we want with transparency. So we will need 2 AO shaders. The first is my regular AO shader, and the second is one with transparency. There is a trick to creating the AO shader with transparency. I create a mental ray material, and in the first slot I insert a mix map. In color #1 I insert a "material to shader". That shader is an A&D material, where I turned up the refraction to 1.0, the IOR to 1.0. For color #2, I use my regular AO shader, and for the mix amount, I use the leaves transparency map to mix the AO with the 100% transparent A&D shader.

When I render my AO pass the easiest way to do this is plugging in the AO material into the material override in the Processes tab of the Rendering Settings. But it's not enough to insert the AO material with the transparency into the material override, because we also have regular objects that shouldn't render with the leaves transparency. To overcome this problem I create "Sub-Object" material shader, with ID 1 being the regular AO shader and ID 2 being the AO shader with transparency. I plug the Sub-Object material into the material override.

Then I select all of the objects in my scene (in this case it's the tree trunk), and apply a material modifier to assign the material ID to 1. Then I select my leaves in the scene and also apply a material modifier, and set the ID to 2.

Presto, the scene is ready for an AO pass with transparencies!

If you have several different transparency maps in one scene, you will have several AO shaders for each transparency, and put them all into the Sub-Object material, then you will need to apply the material modifier with the correct ID to each object for it's respective transparency maps.


  1. Awesome. Once again, very useful tips.
    I'm doing a scene of a necklace falling in a glass of water. It involves a small movement on the surface of the water and a lot of air bubbles inside the liquid to represent the action.
    Can you help me out with this, maybe post some tips? I'm having a hard time with the water, specially the bubbles and drops. Also, it's hard to make the metal look realistic inside the liquid. =]
    Any help will be appreciated!
    Thanks a lot!


  2. You just saved me a lot of troubles. Thank you a lot!! :) Great blog!

  3. i had following all the step but it didn't work!
    i'm working with Max 2010.

    by Joe

  4. be sure your exposure is turned off

  5. hi, nice one.
    I also use this trick, but instead of using the material to shader with an A&D material, it might be easier to use the transmat shader. transmat is a simple "do nothing" shader.

  6. hi, nice one.
    I also use this trick, but instead of using the material to shader with an A&D material, it might be easier to use the transmat shader. transmat is a simple "do nothing" shader.

  7. Thanks for the tip, unfortunately its not working properly for me. I've managed to get the AO render going, but am unable to see "through" the leaf plane... All I get is a nice AO leaf in a black box that blocks out anything below it.

  8. Hate to bother you as I'm sure you're pretty busy. But I have a quick question about this AO method.

    Is there a way in 3DS Max to make the AO shadow work with the transparency as well? I've noticed in my tests that even though the surface is "cut out" correctly, the underlying AO shadow is still the shape of the original geometry.

    In other words, a leaf transparency will show the white surface as being correctly cut out, but the plane geometry is still causing a plane-shaped shadow underneath.

    Many thanks!