Post-processing effects in Leadwerks 5

[Josh]

I am starting work on the new post-effects system and am creating this topic for discussion.

The system works by loading a JSON file that defines a shader, adjustable shader settings the user can modify, and which textures are required. In the new system you do not have to blur images yourself. Instead just set "requireBlurredColor" to true and you will get a blurred image of the previous color texture in the effects stack. This was a problem in some of @shadmar's great effects even, because a lot of his scripts did not account for a changing buffer size.

The system works like this:

auto fx = LoadPostEffect("PostEffects/Bloom.json");
camera->AddPostEffect(fx);

There is no setting a shader or script as a post effect. There are no scripted post effects at all, because no Lua code runs in the rendering thread.

"Bloom.json" will look something like this. Note that no vertex shader is defined:

{
    "postEffect":
    {
        "shader": "Shaders/PostEffects/Bloom.spv",
        "requireBlurImage": true,
        "parameters":
        [
            {
                "name": "overdrive",
                "value": 1.2
            },
            {
                "name": "cutoff",
                "value": 0.15
            }            
        ]
    }
}

The only new effect I can think of that I want to add is per-pixel motion blur. In addition we should have:

• Bloom / Iris adjustment
• DOF (near and far, with adjustable ranges)
• SSAO (still useful for fine details)
• Godrays (may work for all lights, not just directional. I think I can use a raytracing approach for this.)

I think I will skip SSR because the voxel raytracing replaces this and provides more consistent results. There are lots of other little effects that could be added like film grain, grayscale, etc. but I think this is best left as an external pack of shaders made by me or other developers.

Distance fog is built into the main shaders and does not require a post-processing effect.

I might actually remove the camera SetGamma command and merge this with some kind of tone mapping effect.

I think bloom, tone mapping, and SSAO will be enable in all created cameras, by default. In fact these can probably be combined into a single shader.

Am I missing anything else important? Is there an another effect I should add, or a problem I have not accounted for?

[Thirsty Panther]

Shadmars underwater shader was popular.

Only other one I can think of is the radiation shader.

[Mattline1]

Cool, sounds like it will be easier to set up simple post processing effects. Will it be as flexible as the old system? My main concerns would be with ordering of post effects and how modifiable default effects and settings are.

A few questions:

• Could default post effects be disabled or modified, in case the defaults don't match the art style of a project.
• Can the amount of blurring be changed?
• Does the blur just occur on the color map? depth too?
• Is there still a way to manually blur? Box blur looks very different to a guassian, and both have uses.
• Are shader settings modifiable at runtime?
• If bloom, tone mapping, and SSAO are baked into the camera what happens if another post process effect needs to occur in between them, i.e SSR needs to happen before tonemapping.

I guess if there's no Lua code running in the rendering thread it will never be as flexible, but the LE4 system was very powerful as a result of it's ability to be heavily controlled via scripting.

Being able to manually modify default shader source would answer a lot of these questions, is that still possible?

 

 

[gamecreator]

7 hours ago, JMK said:

adjustable shader settings the user can modify

Do you mean that these will be able to be changed by code?  Like you could create an effect where you blur the screen more and more as someone gets more and more drunk or dizzy?  Or use it as a fade in/out effect for menus?  I'd love for this to be the case in the new engine.

Quote

Am I missing anything else important? Is there an another effect I should add

Yes!  Thank you for asking.  A shader you need to project a camera view onto a surface, like a TV or a screen.  I was just researching how to do this and saw on the forums other people trying to do it.  If you know shows like Star Trek, I was thinking of putting on screen the view of a camera in space looking at a ship:

Or use the same trick to show someone talking to you:

Other than that, whatever helps make the games look beautiful, which I know you're already working on.

[Josh]

Yes, that should be possible. Instead of using shader uniforms you will just have access to a set of up to maybe 16 float values, or something like that, and the JSON strings will give them names for identification. So you could call something like effect->SetProperty("strength", 0.5).

You don't need a shader to render to a texture. There is a command called Camera::SetTarget() or Camera::SetRenderTarget (check the docs) that does this.

Depth textures never get blurred. You can't blur them and get useful results.

I think the default post effects will be indicated in a settings.json file. The engine will just by default load up whatever effects that file indicates. You can call Camera::ClearPostEffects() and then add your own shaders, or modify the settings file. The point is, I know if I don't have some default settings then 90% of screenshots in the new engine will have no post-processing effects and it will look bad.

Post effects are drawn in the order they are added to the camera. You can use separate effects. I am just saying if you know you want bloom, SSAO, and tone mapping, it is more efficient to just pack it all into one single shader rather than drawing it in several passes.

The default blur will just downsample the image several times, down to something very small like 1x1. So you could access whatever blur level you want. Maybe level 4 for bloom and the last level for iris adjustment. The only downside is customized blurring would not really be possible. With this design I don't see any way to make that horizontal blur bloom effect (I guess it's called "anamorphic"):

Still, that is a fairly edge case. It would be easier for me to just hard-code an additional blur option than to implement some complicated flowchart scheme. 

Are there any other weird cases where a linear progression drawing from one buffer to the next one isn't going to be sufficient? Are there any other weird custom blurs I should be thinking about, or anything of that nature?

Hmmm, perhaps the answer is not a flowchart, and not hard-coded options, but a series of subpass settings in the JSON file, something like this:

{
    "postEffect":
    {
        "buffers":
        [
            {
                "width": 0.5,
                "height": 0.5
            },
            {
                "width": 0.5,
                "height": 0.5
            },
            {
                "width": 0.25,
                "height": 0.25
            },
            {
                "width": 0.25,
                "height": 0.25
            },
            {
                "width": 0.125,
                "height": 0.125
            },
            {
                "width": 0.125,
                "height": 0.125,
                "index": 0
            }
        ],
        "shader": "Shaders/PostEffects/Bloom.frag.spv",
        "subpasses":
        [
            {
                "source": "COLOR",
                "target": 0,
                "shader": "Shaders/PostEffects/Utility/hblur.frag.spv"
            },
            {
                "source": 0,
                "target": 1,
                "shader": "Shaders/PostEffects/Utility/vblur.frag.spv"
            },
            {
                "source": 1,
                "target": 2,
                "shader": "Shaders/PostEffects/Utility/hblur.frag.spv"
            },
            {
                "source": 2,
                "target": 3,
                "shader": "Shaders/PostEffects/Utility/vblur.frag.spv"
            },
            {
                "source": 3,
                "target": 4,
                "shader": "Shaders/PostEffects/Utility/hblur.frag.spv"
            },
            {
                "source": 4,
                "target": 5,
                "shader": "Shaders/PostEffects/Utility/vblur.frag.spv"
            }
        ],
        "parameters":
        [
            {
                "name": "overdrive",
                "value": 1.2
            },
            {
                "name": "cutoff",
                "value": 0.15
            }            
        ]
    }
}

 

[Mattline1]

4 hours ago, JMK said:

a series of subpass settings in the JSON file

That sounds like a great solution. It would cover all seperable filters.

4 hours ago, JMK said:

re there any other weird cases where a linear progression drawing from one buffer to the next one isn't going to be sufficient?

Temporal anti-aliasing? That would need the current frame and a re-projected frame based on previous frame and a velocity buffer.

[gamecreator]

5 hours ago, JMK said:

Yes, that should be possible. Instead of using shader uniforms you will just have access to a set of up to maybe 16 float values, or something like that, and the JSON strings will give them names for identification. So you could call something like effect->SetProperty("strength", 0.5).

That would be a great step toward flexibility but I wonder: is this not something that can be manipulated in memory?  Can you alter shaders after you load them from a file?  Guessing that you can't from what you're saying.

5 hours ago, JMK said:

You don't need a shader to render to a texture. There is a command called Camera::SetTarget() or Camera::SetRenderTarget (check the docs) that does this.

Oh man.  Didn't even know this was an official function.  Testing it now... thanks!  Edit: it works!  Yay!

[Josh]

7 hours ago, gamecreator said:

That would be a great step toward flexibility but I wonder: is this not something that can be manipulated in memory?  Can you alter shaders after you load them from a file?  Guessing that you can't from what you're saying.

Define "alter"? You mean change the source code? Shaders in Vulkan are loaded from compiled SPV files. This is nice because there is never any question whether a shader will compile successfully on one card or another. It would be possible to integrate the shader compiler code into the engine, but that's not really something I see a need for.

[Josh]

7 hours ago, Mattline1 said:

Temporal anti-aliasing? That would need the current frame and a re-projected frame based on previous frame and a velocity buffer.

I believe the subpass approach described above would cover that case, because you can copy the contents of the framebuffer to another image and then access that image the next frame.

[shadmar]

Looks like a good approach, I guess one type of blurring will be enough, but I cannot think of a good reason to use any other type than downsizing.

What about matrices? Can you access them like before? projection, camara and normal matrix.
Also getting buffers like normal,position,depth.

[Josh]

56 minutes ago, shadmar said:

What about matrices? Can you access them like before? projection, camara and normal matrix.
Also getting buffers like normal,position,depth.

I am working out the details now, but that should be doable.

[gamecreator]

9 hours ago, JMK said:

Define "alter"? You mean change the source code? Shaders in Vulkan are loaded from compiled SPV files. This is nice because there is never any question whether a shader will compile successfully on one card or another. It would be possible to integrate the shader compiler code into the engine, but that's not really something I see a need for.

Say a game is a fixed 60fps and you want to add that blur effect I talked about over 5 seconds.

effect->SetProperty("strength", bluramount);
bluramount += 1.0 / 60 / 5;  //  Adds a fraction of blur per 60fps frame and 5 seconds

Edit: Actually, isn't this what SetFloat does now?  I don't know much about shaders but would this be a uniform?

[Josh]

17 hours ago, gamecreator said:

Say a game is a fixed 60fps and you want to add that blur effect I talked about over 5 seconds.

effect->SetProperty("strength", bluramount);
bluramount += 1.0 / 60 / 5;  //  Adds a fraction of blur per 60fps frame and 5 seconds

Edit: Actually, isn't this what SetFloat does now?  I don't know much about shaders but would this be a uniform?

Yes, that is the same functionality, although I don't think you have have a global variable like that in a shader, unless you use atomics, which you almost certainly do not want to do.

It's also best to do things on a time basis. You will always get the current time in the shader, but setting variables each frame can be problematic because for every game update, you might have 2-3 frames rendered in the rendering thread. So values changing smoothly would not be exactly smooth. It's better to set a start and stop time, and let the shader do the interpolation based on the current time, which is updated each render frame.

[gamecreator]

6 hours ago, JMK said:

It's also best to do things on a time basis.

That was an issue in the past.  Using GetSpeed isn't accurate so your game won't play the same on different computers.

 

[Josh]

The new engine uses a fixed time step in the main loop.

[Josh]

It took three days but I've got a basic post-processing effect working with Vulkan now: ?

[Josh]

I've got SSAO working in Vulkan now.

[Josh]

Beta is updated now:

 

[thinkappleten]

The FBX model importer will now auto-generate materials for imported models, and select the appropriate shader to use. This even works with animated models that use GPU skinning, so it's super easy to get new models into Leadwerks. Other enhancements include support for multi -track animations and animation sequence names. shareit vidmate

[qcrocknet]

Hello,

Will this be supported when in VR also with Leadwerks 5?

Thank you

[Josh]

17 hours ago, qcrocknet said:

Hello,

Will this be supported when in VR also with Leadwerks 5?

Thank you

Yes.

[mandelvid]

sounds like it will be easier to set up simple post processing effects. Will it be as flexible as the old system? My main concerns would be with ordering of post effects and how modifiable default effects and settings are. vidmate.app  saveinsta 

[Alienhead]

toon shader / post effect? 

[Josh]

9 hours ago, mandelvid said:

Will it be as flexible as the old system? My main concerns would be with ordering of post effects and how modifiable default effects and settings are.

Ordering them is no problem. I have a plan to add customizable settings per effect per camera, but one thing at a time.