The purpose of this writeup is to look at mostly the Pathtracing Kernel settings and see how to fiddle with them to speed up render and iteration times. The section up front discusses a high-level overview of the steps to take so you can quickly refer back, and then the section titled The Nitty Gritty goes into detail about everything mentioned in the first section.
This guide was written using the Cinema 4D version of Octane, but the settings should be the same no matter what DCC you're using (or standalone, for that matter).
The Workflow: High Level
Don't worry if this doesn't all make sense yet, that's what the rest of this document is for.
1. Setup: Switch to Pathtracing. Start with a good custom preset. Low samples (128 or 256), high depth bounces (16/16/8), low GI clamp (16), no fancy post-production stuff like denoiser yet, everything else at default.
2. Build: Put your scene together - keep the Live Viewer off for the most part, unpausing it every now and again to see what you're doing.
3. Rough Pass: Once the scene is pretty much in place and you know what kind of lighting and materials you have going, adjust the kernel settings. Raise the samples a little to get a better idea of what the scene looks like, lower the depth bounces until it starts to affect the reflections/refractions/sss, adjust the GI clamp (see if you can get away with 1 or 2). Use the render region in specific areas if your scene is complex and you want to investigate the effects of the settings on just one area. At this point your image should look like what you want while still rendering in a reasonable amount of time, but it may still be noisy.
4. Tweak: If there are weird artifact issues that aren't being resolved, check the Other Tweaky Settings section of this doc to see if there are specific settings to adjust.
5. Cheat with AI: Try the Denoiser and/or upsampler - make sure you have enough samples to create enough data for them to work properly. It may or may not help depending on your scene. Also start to put in Post effects if needed like Bloom/Glare.
6. Refine and Finish: Slowly bring the samples up (or down if the denoiser and/or upsampler is doing a bang-up job) until you find a balance of image quality and speed you're happy with. Spend more time refining if you have an animation - every second you knock off is multiplied by the number of frames you have to render. Don't worry about it as much if you're doing a still - just get in the ballpark and probably overshoot on the samples, because another it would have taken another minute of tweaking to save 10 seconds of render time, and that's just not worth it.
The Nitty Gritty
Now that you have an overview of the process, let's dive in and explore the settings more.
Kernels
Kernels are methods Octane uses to render scenes. As of this writing, there are three main ones - Direct Lighting, Pathtracing, and PMC. There are also Info Channels (Color, Alpha, etc) which are mainly for compositing which probably needs its own writeup. We're going to focus on Pathtracing in this guide. What's cool is that many of the Pathtracing settings are also in Direct Lighting and PMC and work in similar ways, so this will make it easier to pick those up later as you see a need for them.
You can switch kernels in either the settings (the light gray bar at the top of the Kernel section is actually a dropdown), or in the Live Viewer (the dropdown on the right that usually says either DL or PT).
Kernel Settings
I'm going to split these up into three groups. First are the really important ones in any scene (Max samples, Depth, GI Clamp). These have the most detailed explanation. Then I'll list out secondary settings in the Other Tweaky Settings section that only apply to certain scenes (Ray epsilon, Coherent ratio, etc). Lastly there's a group of settings I'm not going to cover that don't directly apply to render speed/quality (AO distance, Light IDs, etc).
Quick note: In other engines with unified sampling like Redshift, using powers of two (2, 4, 8, 16, etc) is more efficient than arbitrary numbers (19, 1001) and makes the GPU happier. I'm not sure if this is the case in Octane, but I'm very much in the habit of it, so that's why you'll see numbers like 256 and 4096 here.
Max Samples
Max samples is probably the most important setting when it comes to quality vs. speed. A sample is a drawing pass that creates image information. If you think of it like a traditional pencil drawing, the very first pass would be a super quick, rough bunch of sketchy lines that pretty much just defines the shape and location of the objects in the composition. You might be able to tell what the drawing will be, but it's not going to look great. The more passes the artist does, the more defined the image gets, but the longer it takes. After a few hundred passes, the drawing will start to look photoreal. Same idea with samples.Similarly, if you have a few objects with simple materials being lit just by an HDRI, you may be able to get away with even 4 or 16 samples and it'll look good, and it might only take a few seconds. If you have a large scene with a lot of glass with scattering or dispersion, and complex lighting, you're probably looking at several thousand samples, each one taking several seconds by itself to calculate. 256 samples in this complex scene would take longer overall to render than 8192 samples in the simple one. Point is, don't get hung up on the number - it's only relative to the specific scene you're working on.
The default number of maximum samples is 16,000. For general purpose use, you're rarely going to need that many, and it makes things super slow. I think (?) the idea here is that that you can watch the number of samples rack up in the Live Viewer, and decide where to stop it when it looks good enough. The problem is, if you don't change it before you go to render to the Picture Viewer, it's going to render all 16,000 of those samples when a couple hundred would have sufficed. It'll just keep refining and refining the image, long after the point where it won't make a difference to your eye. An hour and a half later you'll start to wonder why people were raving about GPU rendering.
A better strategy is to set the samples low in the beginning of your process just so you get a good enough idea of how the scene is progressing. Then change all the other settings which have a large impact on how many samples you'll need, then come back to the samples and raise them until the scene looks good enough.
Diffuse Depth
Diffuse, Specular and Scatter Depth have to do with how many times a light ray will bounce off, or inside an object. These can drastically impact the look and speed of the render. It's actually a good strategy to keep these numbers too high in the beginning, and then lower them as your scene becomes more complete, because they can completely change the look of your scene if they're too low, and that might lead to hours of trying to adjust materials or lighting to improve your render when all you really needed to do was increase one of these sliders.
Specular Depth
This one controls the amount of light bounces off of, or through specular type materials. This can really impact the look of your image, particularly if you have a lot of things reflecting each other or a lot of overlapping glass. If this number isn't high enough, 4th, 5th, 6th... etc level reflections will just end up being blacked out.
As you can see above, with one bounce, nearly all the reflections are blacked out. With two, most of them are. By 4, it's probably looking acceptable, and at 8 and above it looks great. Start at 8 - if the blacked out reflections are noticeable, start cranking it up. If it looks fine, start lowering it until it doesn't, and there's your threshold (until you add more things in to reflect).
With refractions in a Transmissive material (like glass), there's a pretty massive difference in the look between even 8 and 16 bounces. 8 is barely usable in this case. You can go even higher than 16 (and you may need to) if you have a whole lot of glass shards in your scene.
Scattering Depth
As the name implies, you mess with this when you are using a Scattering Medium in a material, or a fog volume. It doesn't affect the scene if you have no scattering or fog, so you can leave it at the default in this case and it won't add any extra time.
GI Clamp
GI stands for Global Illumination, and that once again refers to how much light bounces around a scene. This one is pretty technical, but the default setting of 1,000,000 is usually way too high, and it can have a pretty substantial impact on render speed. Typically you can get away with a value of 1 (yes, one), but sometimes you have to go up a bit higher, but often not higher than maybe 32.
As you can see, it's really noisy in this scene when the GI Clamp is at the default of one million, but as you reduce it into the realm of reasonable numbers, it gets better and better, until it's pretty clean at 1 (with the same number of samples!). Values from 100-1,000,000 looked about the same for this particular scene.
So why isn't it always set to 1? Well, because certain effects - especially the ones with lots of secondary light bounces - suffer pretty badly when this is set too low.
Here's the effect that just the GI Clamp has on a Scattering Medium material where light bounces around inside the object. It loses a lot as it drops below 8, but then the difference between 8 and a million is pretty much nothing in this scene. As a general rule, Start the GI Clamp at 16, and then almost always adjust it down, especially if you see lots of speckly noise. If you're doing a lot of scattering or refraction, it'd be worth seeing if bringing it up to 32 or so helps the look, and then just use a different method (hot pixel removal or denoiser) to address the noise this introduces.
Hot Pixel removal
Other Tweaky Settings
Presets
Now you've got a good group of settings to start every scene with, and you don't want to have to keep redoing that for every project. No problem, you can save your presets!
I usually make a preset called "good starter set" or something. The only kernel settings I change are the Kernel (Pathtracing), the Max Samples (128 or 256), and the GI Clamp (16 or 32 - I go back and forth on this, but it doesn't make a big difference). I also like my default environment in the Settings>Env tab to be black. Otherwise the defaults are fine. In Cinema 4D, if you want this preset to be the default every time you open the program, you have to make it part of a new.c4d file
Camera Imager Settings (fun with AI)
Now that we understand how to use the kernel settings to draw our pixels in as little time as possible, we can move on to the fun stuff - cheating with Artificial Intelligence! Octane has a truly awesome denoiser, and an upsampler. These three things can shave minutes or hours off your renders and they'll still look great. That said - it's not a magic bullet (yet). You still need to understand what the Kernel settings do in order to give these AI tools a nice solid foundation, and there are actually times where these things will make things slower and worse if you don't know what you're doing. Help them help you!
Denoiser
Octane's denoiser does a fantastic job. It can allow you to drastically lower your samples (and therefore render time), and it can clean the image right up like magic.In C4D's Render settings (not Octane Settings), in the Octane Renderer section, there's a checkbox for Use denoised beauty pass. If you aren't using Render passes to composite later, you're going to want this on, otherwise your image will be denoised in the Live Viewer, but not in the Picture Viewer when you do your final render. This setting will override the denoiser settings, so even if that's off in the Octane settings, if you turn Use denoised beauty pass on, you're going to get a denoised render.
Enough chatter, let's see what this thing can do!
Scattering is pretty GPU-intensive, and using the same settings, you can see that it needs a lot of samples, and therefore time, before it gets smooth on its own. 3 minutes gives a barely acceptable image. With the denoiser, 256 samples is totally fine in 27 seconds.
Here's an object with diffraction. This is also a very GPU-intensive effect. Once again, the denoiser creates usable results in a much faster timeframe, but if you look up close, you can see that the denoised versions blur a bit and start to lose some of the fine detail. This is one of the caveats to the denoiser. It can't create information out of nothing, so for every pixel it builds, it has to look around at its neighbors and copy their homework. The result is that detail can get lost. Whether it's worth the speed increase is up to you.
Denoising isn't instant - there's some overhead that it introduces. In the scene above, you can see that on a Mac running an i7 and Vega 20, denoising actually adds a considerable chunk of time - almost 40 seconds! If you can get a clean render in under 40 seconds on that card, it doesn't ever make sense to denoise. Even with fast hardware, 3 seconds of overhead when added to a 2 second render with 16 samples still adds up to the same 5 seconds that a 128 sample render with no denoising gets you.
In summary, the Denoiser is a great post-production tool to help bring your render times down and clean up your images faster, but you need to have your foundation set (a.k.a. get your Kernel settings dialed in) to get the most out of it.
AI Upsampler
This is a relatively new tool. It works kind of like Alien Skin Blowup or Topaz Gigapixel where it renders at a lower resolution (half or quarter), and then upsamples it right at the end. Like Denoising, it needs a good foundation, and works well on certain types of scenes (ones that don't need to be tack-sharp).
Where the upsampler really shines is in allowing you to use far more samples, and still get faster results. In the example above, it took 4096 samples to get a clean image without the upsampling or denoising. Turning on upsampling allowed for doubling the samples to 8192 while still halving the time, and then adding denoiser on top of that created a result that would hold up well in many situations without adding much more overhead.
It's worth playing with both of these things alone or together just to see if you can shave off time from your renders. Sometimes the results are amazing.
Conclusion
This is a lot of information, but with some practice and knowing where to prioritize your effort, it should help you get you going on a good workflow which will produce faster and better renders! If you're interested in learning more about the Universal Material, check that guide out here. If you want to know some best practices for getting Substances into Octane, I wrote a guide on that as well.