Some PostEffect-System suggestions

[klepto2]

While experimenting with the posteffect system i found some smaller things which would be nice to have:

1. More possible paramters: Currently it is only possible to pass (at least if i look into the shader files) 8 different float parameters to the shader. this might be enough for most simple cases, but for more complex scenarios this might be a show-stopping limitation. eg: 8 floats means just 2 vec4 , which means just 2 colors may be passed or 2 rgb colors with 2 additional parameters.

[Disclaimer] I know that you already use the maximum supported sizes (damn you AMD * ) but maybe there is another way for this (Registering uniform buffers for the rescue?).

2. currently it is only possible to create additional textures with a multiplier of the current framebuffer size, i would suggest to add the possibility to add an absolute size as well. Some effects might need some smaller textures like 256*256 or 32*32 to calculate additional data.

3. Make some passes one time only, with the possibility to mark them as dirty from the api.

Explanation:  Lets say we want to have simple noise distortion pfx which needs a noise texture generated (with a size of 256*256 see pt. 2) now in pass 1 you would generate the noise and use it in pass 2. Currently the first pass is executed everytime, even it is only needed once, but it should be only regenerated, when eg: the seed has changed.

 

* AMD allows just 128 bytes for push-constants while NVIDIA has changed to 256 on more recent cards.

[Josh]

I think the Vulkan spec states that 128 is the minimum. I haven't really looked at how this varies across different hardware.

[klepto2]

yes, 128 is a minimum, but most amd hardware still uses this minimum, while most nvidia cards use a limit of 256: 

https://vulkan.gpuinfo.org/displaydevicelimit.php?name=maxPushConstantsSize&platform=all

[Josh]

#1 is resolved, new limit is 64 floats.

[Josh]

Regarding the others, I think there is a mechanism in place that can accomplish this. If you create another ortho camera and call SetRealtime(false) on it, you can then call Camera::Render once to draw it it once, make it render to a texture buffer, and now you’ve got your static texture.

[klepto2]

yes, this might work. But wouldn't that mean, that we need a way to assign the textures via code? 

From my understanding the current system (based on the json) would still rerender the textures every time. And we don't have a way to get the correct texture index, so we can not pass it with the parameters.  

[Josh]

cam->SetPostEffectParameter(effectindex, paramindex, texture)

The camera used for the special pass can be set to only draw once using SetRealtime / Camera::Render.

[klepto2]

I have tried it this way but none way is working. Any idea why? : 

first i create the textures: i verfied that they are created by applying a material with the texture to a box.

now i tried to pass them to the posteffect like this:

	void Environment::SetupCamera(shared_ptr<Camera> camera)
	{
		_cameras.push_back(camera);
		auto pfx_id = camera->AddPostEffect(_posteffect);
		_camera_posteffects[camera] = pfx_id;	
    }

...

for (auto c : _cameras)
		{
			c->SetPostEffectParameter(_camera_posteffects[c], 3, _textureStorage->m_transmittance_array[READ]);
			c->SetPostEffectParameter(_camera_posteffects[c], 4, _textureStorage->m_scattering_array[READ]);
			c->SetPostEffectParameter(_camera_posteffects[c], 5, _textureStorage->m_irradiance_array[READ]);
		}

in the post effect itself i tried different different ways to get the correct texture, but both failed:

mat4 params = ExtractPostEffectParameters(0);

int transmittance_id = floatBitsToInt(params[0][3]);

col = texture(texture2DSampler[transmittance_id], uv).xyz;

Direct using the PostEffectTextureID which also fails:

col = texture(texture2DSampler[PostEffectTextureID3], uv).xyz;

[Josh]

Your first attempt above looks correct to me but I have not tested it out much...

[klepto2]

Ok, for an experiment, I made everything public in texture.h and corrected my code to add a RENDER_HOOK to assign the textures. (Before rendering the id system is not initialized) 

With these changes the following works and the correct ids are stored in the parameters:

void BeginEnvironmentHook(const UltraEngine::Render::VkRenderer& renderer, shared_ptr<Object> extra)
	{
		auto env = extra->As<Environment>();
		if (env != nullptr)
		{
			env->StartInternal();
		}
	}
  
  void Environment::StartInternal()
	{
		for (auto c : _cameras)
		{
			// First 3 Values are to check if the new parameter system works for simple values and they work;
			c->SetPostEffectParameter(_camera_posteffects[c], 0, 1.0f); 
			c->SetPostEffectParameter(_camera_posteffects[c], 1, 0.0f);
			c->SetPostEffectParameter(_camera_posteffects[c], 2, 0.0f);
			// Made everything public in Texture.h to access the internals and it works as expected
			c->SetPostEffectParameter(_camera_posteffects[c], 3, _textureStorage->m_transmittance_array[READ]->rendertexture->core->id);
			c->SetPostEffectParameter(_camera_posteffects[c], 4, _textureStorage->m_scattering_array[READ]->rendertexture->core->id);
			c->SetPostEffectParameter(_camera_posteffects[c], 5, _textureStorage->m_irradiance_array[READ]->rendertexture->core->id);
		}
	}

but this is not working even with a hook:

	void Environment::StartInternal()
	{
		for (auto c : _cameras)
		{
			// First 3 Values are to check if the new parameter system works for simple values and they work;
			c->SetPostEffectParameter(_camera_posteffects[c], 0, 1.0f); 
			c->SetPostEffectParameter(_camera_posteffects[c], 1, 0.0f);
			c->SetPostEffectParameter(_camera_posteffects[c], 2, 0.0f);
			// Default behaviour with protected access modifiers and it doesn't work
			c->SetPostEffectParameter(_camera_posteffects[c], 3, _textureStorage->m_transmittance_array[READ]);
			c->SetPostEffectParameter(_camera_posteffects[c], 4, _textureStorage->m_scattering_array[READ]);
			c->SetPostEffectParameter(_camera_posteffects[c], 5, _textureStorage->m_irradiance_array[READ]);
		}
	}

parameters for values 3,4,5 are zero.

[Josh]

I might have found the cause just by looking at the code:

	void RenderCamera::SetPostEffectObject(const int index, const int offset, shared_ptr<RenderTexture> value)
	{
		posteffectsobjects[index][offset] = value;
		if (value)
		{
			posteffectsparameters[index][offset] = value->core->id;
		}
		else
		{
			posteffectsparameters[index][offset] = -1;
		}
		UpdateData();
	}

value->core->id should be wrapped in a call to IntBitstoFloat().

[klepto2]

ah ok, so it should even work without using hooks as the id is resolved already when needed. 

but why does it work seamingless with this:

 

c->SetPostEffectParameter(_camera_posteffects[c], 3, _textureStorage->m_transmittance_array[READ]->rendertexture->core->id);

Edit: Nevermind, found it. It is an override for int and i assume this already use the inttofloatbits.

[Josh]

Yeah, if you input an integer, it will automatically convert the bits to a float....in the 32-bit float build only.

The 64-bit build will store them as doubles, since there is no issues with the precision.

[klepto2]

Ok, i will temporarily use the "make public approach" and i get some very nice results

precomputed atmospheric scattering as a posteffect in Ultraengine:

And this is the actual code in main.cpp to integrate it:

 auto settings = EnvironmentSettings::GetEarthSettings();
    auto env = CreateEnvironment(world, settings);
    env->AssignToCamera(camera);
    env->Start();

 

 

[SpiderPig]

So pretty. 

[Josh]

[klepto2]

21 hours ago, Josh said:

I might have found the cause just by looking at the code:

	void RenderCamera::SetPostEffectObject(const int index, const int offset, shared_ptr<RenderTexture> value)
	{
		posteffectsobjects[index][offset] = value;
		if (value)
		{
			posteffectsparameters[index][offset] = value->core->id;
		}
		else
		{
			posteffectsparameters[index][offset] = -1;
		}
		UpdateData();
	}

value->core->id should be wrapped in a call to IntBitstoFloat().

Works now with the latest version

[klepto2]

On a side note, which i will later post in a block in more depth: 

While the limit of parameters is now 64, this is still not enough for the precomputed scattering. I needed at least 88 unique identifiers (maybe 2 or 3 less, as i haven't fully optimized each vec4) + 4 texture slots. so i currently need a  total of around 92 floats/values passed to the posteffect. Well to overcome this i used a common practice used in shadertoy and maybe this can be made more practical later. 

As a background: I need to precalculate a lot of these shadervalues, but once they are calculated, most of them don't change anymore. So i created a tiny texture, lets say 13 * 3 PX in RGBA32 format and the corresponding pixmap.  Then I store each set of Vec4 in the pixmap and upload it to the texture. 

In the shader i can now simply get the values with a texelFetch.

 

void WritePixel(shared_ptr<Pixmap> pixmap, int x, int y, iVec4 data)
	{
		pixmap->WritePixel(x, y, Vec4(data.x, data.y, data.z, data.w));
	}

	void WritePixel(shared_ptr<Pixmap> pixmap, int x, int y, Vec4 data)
	{
		pixmap->WritePixel(x, y, data);
	}

	void WritePixel(shared_ptr<Pixmap> pixmap, int x, int y, Mat4 data)
	{
		pixmap->WritePixel(x, y, data[0]);
		pixmap->WritePixel(x + 1, y, data[1]);
		pixmap->WritePixel(x + 2, y, data[2]);
		pixmap->WritePixel(x + 3, y, data[3]);
	}

	

	void AtmosphereCalculator::updateUniforms_forpfx(double* lambdas, double* luminance_from_radiance)
	{
		auto pixmap = CreatePixmap(iVec2(m_settings_texture->GetSize().x, m_settings_texture->GetSize().y), TEXTURE_RGBA32);
		// constants
		WritePixel(pixmap, 0, 0, _atmosphereData.c.data0);
		WritePixel(pixmap, 1, 0, _atmosphereData.c.data1);
		WritePixel(pixmap, 2, 0, _atmosphereData.c.data2);
		WritePixel(pixmap, 3, 0, _atmosphereData.c.SKY_SPECTRAL_RADIANCE_TO_LUMINANCE);
		WritePixel(pixmap, 4, 0, _atmosphereData.c.SUN_SPECTRAL_RADIANCE_TO_LUMINANCE);
  
  		... much more data goes in here
  
  		m_settings_texture->SetPixels(pixmap);
  }
  
  // and in the shader it looks like this:
  void SetupUniformDataForPFX(sampler2D settings)
{
	AtmosphereParameters atmParams;
	AtmosphereConstants constants;
	
	constants.data0 = ivec4(texelFetch(settings, ivec2(0,0),0));
	constants.data1 = ivec4(texelFetch(settings, ivec2(1,0),0));
	constants.data2 = ivec4(texelFetch(settings, ivec2(2,0),0));
	
	constants.SKY_SPECTRAL_RADIANCE_TO_LUMINANCE = texelFetch(settings, ivec2(1,0),0);
	constants.SUN_SPECTRAL_RADIANCE_TO_LUMINANCE = texelFetch(settings, ivec2(2,0),0);
   
  	...
}