[Advice Request] Cubemap capture and custom render pass shader groups

[Vladimir Sabantsev]

Hello to all the inhabitants of the forum!

I'm experimenting with the third-person camera view in terms of usability for tight scenes with varying height levels and trying to achieve a very specific per-pixel opacity mask filtering. I've had a prototype made in Ultra Engine (the other one) and managed to obtain the basic level of what is needed, but lots of workarounds were required along the road and I can see that it just won't work in a good way without editing the source code of the engine due to some higher-level limitations.

1. The main pillar to achieve the intended visuals is to have a real-time cubemap depth capture of the environment around the player similar to what a point light would have. I was not able to find a ready-to-use entity to capture cubemaps in Ultra Engine. From what I see in the documentation, it will be required to create 6 cameras rendering to unique texture buffers before creating the main camera to make a pre-pass cubemap capture of the scene.
   
   
   Is it right that only a depth test will be performed for a TextureBuffer created with CreateTextureBuffer(..., colorattachments = 0, depthattachment = true, ...)?
   Is there a way to simplify the cubemap capture setup by having a shared texture buffer?
   Is there a standardized way to capture cubemaps that I just didn't manage to find?
    
2. The other requirement is to have another pre-pass with the same view frustrum as the main camera, but with a different shader applied to all the entities of the scene.
   I've managed to find this article (Shader Families from 24 July 2019) with some information on ShaderFamily which is not available in the documentation (ohh how good it would have been to have it in the docs). From what I see, there is a possibility to have a different shader for different rendering scenario and rendering pass - great, shadow pass switch that I will need later is crossed out!
   
   
   I assume that a regular camera created with CreateCamera method uses the "base" render pass shader group and to have a different fragment shader applied one might set a different material (shader family) to all the required entities duplicates with a different bit passed to Entity::SetRenderLayers.
   Is there a way to to create a custom render pass shader group and then assign it to a camera instead of the "base" one to use it together with specifying Entity::SetRenderLayers bitmask without the need to duplicate entities?

[Josh]

Both the points lights and the GI probe renderer in the editor render to all sides of a cubemap.

What do you plan to use this functionality for?

[Josh]

It is probably possible to set up six cameras that each draw to a different texture buffer. The texture buffer class can accept an index when a color texture is applied to it, so each texture buffer could render to a different cubemap face:
https://www.ultraengine.com/learn/TextureBuffer_SetColorAttachment?lang=cpp

This functionality is not in the SetDepthTexture command, but I could probably add it without much trouble.

[Josh]

4 hours ago, Vladimir Sabantsev said:

The other requirement is to have another pre-pass with the same view frustrum as the main camera, but with a different shader applied to all the entities of the scene.

What does your different shader output? Sometimes it is easier just to use a post-processing effect or turn the distance fog up all the way. This is how I do the orange outlines in the editor.

[Vladimir Sabantsev]

1 hour ago, Josh said:

It is probably possible to set up six cameras that each draw to a different texture buffer. The texture buffer class can accept an index when a color texture is applied to it, so each texture buffer could render to a different cubemap face:
https://www.ultraengine.com/learn/TextureBuffer_SetColorAttachment?lang=cpp

This functionality is not in the SetDepthTexture command, but I could probably add it without much trouble.

Did you mean to write "It is probably possible to set up six cameras that each draw to the same texture buffer."?
But how would a camera know to which of the attachments to render to? Is there an interface to pass the attachment index? I can see that Camera::SetRenderTarget receives only the TextureBuffer itself and nothing else.

 

1 hour ago, Josh said:

What does your different shader output? Sometimes it is easier just to use a post-processing effect or turn the distance fog up all the way. This is how I do the orange outlines in the editor.

The minimal requirement for this shader is to output a mask which corresponds to all the pixels which are not visible from the players' character position by filtering out all the pixels with the help of the cubemap depthbuffer captured at the players' character position. You can think of it as a mask for all the pixel which receive a shadow from a point light with a huge radius placed inside the players' character.

6 hours ago, Vladimir Sabantsev said:

Additional output which might be helpful is a layer populated with entity IDs for pixels in the masked-out areas to prevent some weirdly looking cases of self-masking on the final image (you can notice that a huge character on the street has an unfortunate accident on the upper part of the head).

1 hour ago, Josh said:

Both the points lights and the GI probe renderer in the editor render to all sides of a cubemap.

I had an idea to use something like this, it can cover the first stage. But is there a way to grab their cubemaps in C++ nowadays? I saw an answer from a far back time saying that it's available as the first texture of entity material, but now I see that Entity::GetMaterial is commented out and is subsequently unavailable for the Light and PointLight classes.

[Josh]

1 hour ago, Vladimir Sabantsev said:

Did you mean to write "It is probably possible to set up six cameras that each draw to the same texture buffer."?
But how would a camera know to which of the attachments to render to? Is there an interface to pass the attachment index? I can see that Camera::SetRenderTarget receives only the TextureBuffer itself and nothing else.

Yes, the color attachment function has an optional index parameter. The depth attachment function currently does not have this parameter.

I will look into adding the cubemap face index for the depth attachment method.