Fog in Leadwerks 4.4

Entry posted by Josh · June 19, 2017

3,458 views

Distance fog is one of the most basic visual effects in 3D graphics, going back to the 1990s. Here is the effect in the Quake 3 Arena map "Fatal Instinct", which was shrouded in a dense orange fog:

Leadwerks Game Engine 2 had this available as a built-in effect, while the more flexible effects system of Leadwerks 3/4 had several Workshop shaders available to use, including one by Klepto and another one more recently added by myself. However, this has not been part of the official SDK until version 4.4. Why is that?

The Problem
When water is rendered in Leadwerks, a low-quality render is performed of the world with the camera scale inverted on the Y axis. Because the reflection is distorted by ripples, we render to a lower-resolution buffer with all settings on low and effects disabled. This is important because it gives a faster performance and image quality that is still acceptable. The water plane also uses occlusion culling so that the extra pass is only rendered if part of the water plane is visible. All post-processing effects are disabled in the reflection pass, which makes good sense, except in the case of fog. If the world is shrouded in dense fog but the reflection in the water is clear, it creates an obvious problem. In the screenshot below, a post-processing effect is applied to the world. Although the water does have fog applied to it, the reflected image on the water does not have any fog, creating a stark problem, because post-processing effects are disabled in the reflection pass.

Pre-Rendering Fog
One option would have been to allow the user to mark some post effects as visible in reflection passes, but that seemed complicated and error-prone. I came up with the idea build a simple fog calculation into the first ambient or directional lighting pass. Here it is applied in the directional light pass:

And here is what happens when the fog effect is applied in the directional light pass in the water reflection as well:

We can modify the water shader itself to add the same fog calculation to the surface of the water. Now our water matches the world.

Removing Artifacts
There was still more to do. We are rendering fog first and other stuff later. Anything rendered after the main lighting pass has to use the fog calculation to substract its effect from the scene. For example SSAO will add shaded areas on top of fog, which we definitely don't want. See the bridge and trees in the distance.

The solution is to add the same fog calculation into the SSAO shader:

	float fogeffect = 0.0f;
	if (fogmode==true)
	{
		fogeffect = clamp( 1.0 - (fogrange.y - length(worldCoord - cameramatrix[3].xyz)) / (fogrange.y - fogrange.x) , 0.0, 1.0 );
		fogeffect*=fogcolor.a;
		if (fogeffect==1.0f)
		{
			fragData0 = outputcolor;
			return;
		}
	}

And then use the fog level to lessen the impact of the effect:

fragData0 = outputcolor * fogeffect + outputcolor * (sumao / float(passes)) * (1.0 - fogeffect);

The underwater artifacts are a separate issue that were solved by adding another calculation. Here is the result:

The same fog calculation had to be added to all light shaders, light volumes, and probes, to make sure lights faded into the fog. Other effects can make use of four new built-in uniforms which will provide all the fog information the shader needs:

uniform bool fogmode;
uniform vec2 fogrange;
uniform vec4 fogcolor;
uniform vec2 fogangle;

On the client side, eight new commands have been added to the camera class to control the fog appearance, which are also available in Lua:

virtual void SetFogColor(const float r, const float g, const float b, const float a);
virtual void SetFogAngle(const float start, const float stop);
virtual void SetFogRange(const float start, const float stop);
virtual void SetFogMode(const bool mode);
virtual Vec4 GetFogColor();
virtual Vec2 GetFogAngle();
virtual Vec2 GetFogRange();
virtual bool GetFogMode();

This was the solution I've had in mind for some time, but I haven't had a chance to implement it until now. It works really well and provides robust fog that looks correct under a wide variety of settings.