Render Series
Version 1.0, Updated November 2024 using Octane 2024.1 and Cinema 4D 2025.0.2
About this Guide
This is the third guide in the Post Production/AOV series. It focuses on setting up object masking using the Cryptomatte and Render Layer systems in Octane. This is useful for both editing specific objects in post, or punching them out to put on a totally different background.
There’s also a bonus section at the end that talks about Affect Alpha and Shadow Catcher - two material properties that help with compositing.
A PDF version of this guide can be found here
Overview
Octane has two current systems and one deprecated one for masking out parts of a render:
Cryptomatte creates masks for the final render that are used to isolate all of the various objects for image correction in post (changing colors, adjusting brightness, etc). It supports things like motion blur and depth of field, making it incredibly useful for masking animations in particular.
Render Layer is more for isolating objects so that we can do a full background replacement in post or export out a transparent PNG for web/app/presentation use. It doesn’t support motion blur or depth of field, but it does give us options to render out shadow layers and reflections for the objects so we can better composite them into the new background.
Render Layer Masks (deprecated, not to be confused with Render Layer) is the older, far slower way of doing what Cryptomatte does. This system has been replaced by Cryptomatte, but still exists for compatibility purposes with older files. It’s highly recommended to update older projects to one of the other two systems if they need to be reused. Confusingly, there’s a Render Layer Mask AOV which is different from the rest of this system and still relevant, but we’ll cover that later.
Render Layer Masks are supported in C4D’s standard EXR format, but not the EXR(Octane) format, and we’ll actually see the entire Render Layer Mask section disappear when we select EXR(Octane) from the Format dropdown, and reappear when we change it to EXR.
The Layer ID system
Both Cryptomatte and Render Layer make use of the Layer ID system to control the grouping of objects we want to mask or isolate. Render Layer relies on this exclusively, while Cryptomatte has other ways of breaking objects apart.
The Layer ID field is found in the Octane Object tag in the Object Layer tab. This allows us to pinpoint individual objects and manually assign them groups for them without having to nest or reorganize them in the Object Manager. There are several types of ID designations in the Object tag. The Layer ID field in the Object Layer tab is what we’re focused on here, not Instance ID or Bake ID.
Important: When assigning Layer IDs for separating objects, we never want to use Layer ID 1. Anything on Layer 1 will be treated the same as the background and any unassigned objects and will not be broken out from the rest of the scene.
Part I: Cryptomatte
Cryptomatte creates several different mattes in the same frame that can be used to break out specific areas of the render to further edit in post. This has several advantages over other systems, such as supporting motion blur, depth of field, and being able to break apart an unlimited number of objects.
Octane for C4D currently supports several ways of assigning mattes to the various objects and selections in the scene. This is chosen in the Crypto type dropdown when we add the pass. There are also a bunch of methods toward the bottom that are for Octane Standalone compatibility only and are unstable in the C4D plugin (so… don't use them as of this writing). Mattes always show up to us as different random colors to make visualizing and working with the system easier.
Material node name breaks apart the objects by the NAME of the material assigned to them. Any objects or selections with the same material applied to them become part of a single mask. The matte colors are randomly generated based on the name, so if we don’t like one, we can change the name of the material slightly (add a 1 or an extra space, etc.) and it will change the matte color to make it easier to see.
Important: If two materials have the same name, even if they’re completely different materials, they will be combined into one matte. This mode requires that the material names are unique.
Material pin name is kind of tricky. This is based around the Selection field in the Material Tag’s Tag tab (tongue twister, that). The material tag needs to have a unique name, AND that name has to correspond to a polygon selection that’s named the same thing. When those two conditions are met, Cryptomatte will give that polygon selection a unique matte.
In the example above, there are two polygon selections. One is called Upper Stripes, and the other is called Lower Stripes. There are two material tags that are linked to the same glossy black material. One of the tags’ Selection name is Upper Stripes, and the other is Lower Stripes (they match the tag names exactly). As a result, both of these have their own unique mattes (magenta and green in this case).
If there’s a polygon selection with no corresponding material, nothing happens.
If there’s no Selection name in the material tag (like the material on Backdrop above), that geometry goes on the default matte (blue in this case).
If there’s a Selection name, but no corresponding polygon selection tag, ALL of the geometry with ANY name goes on the same matte (purple in this case). In the example above, the Selection name for the shell rim, body, and shell base are all called random things, and they all appear on the purple matte.
Object node name uses the Name field found in the Basic properties of the Octane Object Tag to separate objects out into different mattes. If several Object Tags are named the same thing, the objects those are attached to will appear on the same matte. All clones in a cloner appear on the same matte.
Important: If an object has an Object Tag, but we don’t change the name of it, it will go on default matte with anything else that doesn’t have a tag (the shell rim above has the default name “Octane ObjectTag” and it’s on the blue matte, just like the background that has no tag.
Geometry node name uses the name of the object itself (found in the Basic tab of the object’s attributes, or changeable directly in the object manager). If several objects are named the same thing, they’ll appear on the same matte. Each clone in a cloner appears in a different matte.
Render Layer uses the Layer ID field in the Object layer tab in the Octane Object tag. Several objects with the same Layer ID will appear on the same matte. All clones in a cloner appear in the same matte.
User Instance ID is used to break up particle systems and clones into their own mattes.
Important: Any object that does not have an object tag is assumed to be on ID=0, which can cause problems as seen above. By tagging other objects and assigning an Instance ID of a high value out of the range of how many clones or particles we have, we can make sure those objects are on different mattes.
Also important: A cloner in Instance mode needs an Effector with the Color Mode set to Effector or Fields for this AOV to work properly. Multi-instance/Render Instance mode works fine without one.
Using Cryptomatte in Post
Several applications support cryptomattes, and they often handle them in different ways.
Important: In Octane, if we export a Cryptomatte AOV using the Picture Viewer, we need to make sure EXR(Octane) is chosen and not C4D’s stock EXR format. The stock C4D-native EXR format causes problems with Octane and never works right in post apps. For more on exporting, check out the end of the Intro to AOVs guide.
This isn’t an issue if we’re saving directly from the Live Viewer because it automatically saves to the correct EXR format.
Still Compositors
Photoshop: Currently on Windows there’s a terrific solution in the form of a free plugin called Exr-IO. Unfortunately it’s not made for the Mac version of Photoshop, and based on some quick searching around, there isn’t really a viable alternative (if you know of one, please leave it in the comments).
As of this writing, there also doesn’t seem to be a solution for Affinity Photo :(
Motion Compositors
After Effects: https://youtu.be/_cOG53NWJj4
Nuke: https://www.youtube.com/watch?v=3j9VRGhYn9g
Davinci Resolve: https://www.youtube.com/watch?v=_rqp5e-idFk
Autodesk Flame: https://youtu.be/0GAJ_qSrbg4
Octane
Octane itself can use the Cryptomattes it generates in the Output AOV compositing system which we’ll be exploring in the next guide in this series.
Part II: Render Layer
Render layer is a good option if we want to do a full background replacement, reposition our objects in post, or create images with transparent backgrounds to be used on the web or in presentations.
Render Layer basic setup
Normally there would be an important tag here, but all of this needs to be set up properly or it won’t work at all, so it’s all important :)
-
Alpha channel MUST be enabled in the Octane render settings (in Kernel settings) for this to work.
-
At least one object in the scene must have an Octane Object tag and be set to a Layer ID other than 1. If it’s set to 1, it will render with all the other objects that don’t have tags and we’ll end up with unwanted objects in our PNG.
-
Finally in the C4D Render Settings (not the Octane ones), in the Octane Renderer section, in the Render AOV group tab, at the bottom, Render Layer must be enabled, and the Layer ID of the object(s) we want to isolate needs to be set here.
From here, the fastest way to get our transparent image out is to save directly from the Live Viewer. We want to make sure Premultiplied alpha is set to None, but that’s the default, so we should always be fine there, otherwise it’s a pretty painless process.
Saving from the Picture Viewer also works with Render Layer so long as our color management settings match Octane’s.
But wait.. Can’t we just just hide the background from the render and use an alpha channel to get the same effect without setting up the Render Layer system?
Yep, but...
The real power of Render Layer is in the other built-in compositing modes which are a lot easier to wrangle than setting up a bunch of inclusion/exclusion tags, and then the extra shadow and reflection AOVs that we can’t get any other way that help us blend our punched-out objects in with the new background more realistically.
Render Layer Modes
The Mode dropdown in the Render Layer section of the Render AOV group settings controls which pixels will actually render in ALL of the various render layer passes.
Normal Mode takes the rest of our scene into account. This means other objects will be reflected in the objects and affect the lighting of the active Render Layer. If anything that’s on another layer is obstructing objects that are on our active Render Layer, those portions of our Render Layer will not render and we’ll end up with “clipped” sections (see the graphic above).
Hide Inactive Layers will disregard all other objects in our scene and just take the lighting and environment into account. There won’t be any clipped portions of our main object due to obstructing objects, but there also won’t be any reflections or other effects from those other objects either. This is similar, but not exactly the same as if we just turned off Render Visibility for everything else.
Important: The emission from hidden objects still affects the render layer. In the example above, the cat’s body has no color in it all and the HDRI is monochrome. The subtle color in the chest and around the body is actually coming from the emission of the mice, even though the reflections from the mice are not showing up in the coating layer or collar and eyes like they are in the “hide from camera” example.
Hide from Camera takes the rest of the scene into account (other objects show up in reflections and affect lighting), but unlike Normal mode, regardless of whether other objects are obstructing objects on the Render Layer, it will not clip out portions of the render. This is probably the most useful mode.
Only Side Effects Won’t render the beauty pass at all - it will only create data for the Layer Shadows and Layer Reflection AOVs, which we’ll look at next.
Render Layer AOVs
Octane has a set of three AOVs that are specific to the Render Layer system that will enable us to get shadow and reflection data for our active Render Layer, and two utility ones to help visualize the different IDs and create quick masks for us.
Important: This would be a good time to mention that we can only render ONE Render Layer at a time (controlled by the Layer ID), and Octane will only render what is on the active Render Layer. If we are using other passes (beauty, info, etc.), those passes will also only render pixels that exist on the active Render Layer. If we want to automate rendering several Render Layers, we need to do this using the Takes system and set up a different C4D render setting for each Layer ID.
Black Layer Shadows AOV
The Black layer shadows AOV captures all of the shadows that all of the objects on the render layer cast. This needs to be added like any other AOV, viewed using the little orange tabs in the Live Viewer (_LBSh) and exported either through the Live Viewer or via Octane’s Export settings in the Render AOV group tab (not C4D’s Save area on the far left) when using Render to Picture Viewer.
This AOV creates an image with an alpha channel which is super convenient for being able to use directly on the web or in a presentation program or something that doesn’t allow for blend modes.
Theory time! As it says in the name - this pass will only produce black shadows (with varying levels of alpha), even if we’re hitting the object with colored lights or have a colored background. Shadows are an absence of light caused by the object obscuring the light source, so they don’t have a color to them, just different levels of dark depending on the properties of the material and nature of the original lighting. This is why this pass can have an alpha channel where others do not.
Normal mode works the same way it does in the Beauty AOV - anything obscuring the shadows (part of side effects) will be clipped, even things on the active render layer as we can see above.
Only side effects mode is particularly useful with shadow and reflection AOVs because it hides the geometry of the render layer from the camera as well, meaning we get a complete shadow or reflection. This is great if we want to separate the shadow/reflection layer in post (like animate it on, for instance).
The other two modes (hide from camera/inactive layers) are often not very useful with this AOV since they’ll hide whatever background objects are receiving shadows, leaving us with a blank pass.
Layer Shadows AOV
The Layer shadows AOV (not to be confused with Black layer shadows) captures both black shadows - though not on an alpha background - and the fake colored shadows when light is cast through colored transmissive materials.
Why fake? Because physics doesn’t work like this. If we shine light through a transmissive material of any kind or color (glass, diamond, etc), the shadows are always black (or an absence/obscuring of light as we just learned). What produces the coloring on the other end is light scatter patterns, or caustics. Caustics, as we know from the caustics guide, is computationally expensive and doesn’t AOV super well.
One workaround for caustics being so brutal is something known as architectural glass, which is a technique that pulls the transmission color from the material and uses that to cast fake colored shadows instead of light scatter patterns. This is usually seen in - you guessed it - architectural renderings because it allows for at least some sort of color casting/fake caustics from colored glass, even if it’s not physically accurate. This technique is FAR faster than caustics and can pass as realistic-ish if it’s a small part of a larger scene.
In Octane, architectural glass is turned on the Common tab by ticking the Fake shadows checkbox. When that’s on, we can use the Layer Shadows AOV (_LSh in the Live Viewer) to comp the effects of the fake colored shadows in our new background. Of course we could just alter the color of the black shadows AOV in post, but if we have multiple colors of glass, this gets tedious.
So, to get this AOV to work:
- The material needs to have specular type transmission.
- The transmission of the material needs to have some sort of color in it that’s not white.
- A light of some sort needs to shine through the material and cast shadows on a matte material.
- In the Common tab, the material needs to be set to Fake shadows.
- All the other things that affect the other Render Layer AOVs need to be set up right.
Important: Layer shadows AOV doesn’t have an alpha channel like the black shadows AOV, so we need to use a blend mode to composite it into our scene in post (meaning we can’t just produce a transparent PNG for the web easily).
Layer reflections AOV
The Layer reflections AOV picks up the reflections in other objects caused by the geo on our Render Layer. The modes work the same as the other two Render Layer AOVs with inclusion and exclusion of the objects in the scene.
This AOV doesn’t have an alpha channel, so we need to combine it using a blend mode in post. The blend mode depends entirely on the new background and the reflections themselves. The background is always black, so we’ll need to use a mode like Add or Screen - different ones have different effects, so we’ll just need to play around with them until we get the right one.
Important: Unlike the Black Layer Shadows AOV, the Layer Reflections AOV picks up the color of the objects that are doing the reflecting, so most of the time we’ll want to use a neutral background (some shade of gray) if we’re not sure what color the replacement background will be.
Render Layer Utility Passes
The Render Layer ID AOV just shows all the Layer IDs so we can see what our objects tags are assigned to. Again, anything on Layer ID 1 is the same as unassigned objects, so we never want to use that to isolate an object. This is just for visualization, not masking. Cryptomatte set to Layer ID would be a better solution if we want actual masks for all our render layer assignments.
The Render Layer Mask AOV gives us a white mask for the active Render Layer when it’s turned on (ID 2 in the example above). The modes work the same way as the other AOVs and the mask has an alpha channel which is nice. This is a quick way to get a mask for a single Layer ID if we don’t want to deal with Cryptomatte’s setup, or need the mask for the full, unobstructed object by using one of the Hide modes.
Bonus: Affect Alpha and Shadow Catcher
Affect Alpha
Recent versions of Octane allow us to punch out some of the translucent areas of the render for easier compositing. This setting is a simple on/off checkbox in the Common tab of the material called “Affect Alpha”. It’s a material-wide setting, and there are no options for it like threshold or anything else.
Turning this on is easy. Making the final composite look good is not. There are a lot of factors that we have to set up just so to make the final result convincing including the lighting (especially the HDRI), background objects, camera angle, and material properties like IOR, roughness, sheen/coating, specular, transmission, etc. Once that’s all set up to our liking, it might help to include some fake IOR effects in the compositing program.
Shadow Catcher
The Shadow Catcher material setting is a quick and dirty way to include shadows in a render with alpha. It’s applied to a material in the Common tab by turning on “Shadow Catcher”. That material then has to be applied to an object like a plane or a cyc that will actually catch the shadows from our object.
The Material acts kind of like the Black Layer Shadows AOV, but lacks the compositing features of it like Only Side Effects mode. The main advantage of it is that we can just knock out a render with a shadow baked in real quick without having to re-composite the layer shadow in post if that’s all we’re looking for.
Wrap Up
That’s it for masking options in the Render AOV system. At this point, you should have a good idea of what we might use Cryptomatte and the Render Layer systems for, and some general knowledge of how masking works in Octane.
We’ll explore more specific workflows and techniques in future guides, so stay tuned.