Simplistically speaking, you could think of the vertex stage of a shader as the "position" of the object, and the fragment stage as the "color." All you really need is position and color. Vertex displacement manipulates the position to warp or move the object. There are many ways to do it but the whole trick revolves around the ex_vertexposition. You may notice in every model shader you have these two lines in the vertex stage:

   ex_vertexposition = entitymatrix_ * vec4(vertex_position,1.0);

   gl_Position = projectioncameramatrix * ex_vertexposition;

I know from things I've seen around the internet that these two lines are establishing the object in model space and camera space. I don't know much about that but roughly the way I see it the first of these lines establishes what the  ex_vertexposition is, and the second line outputs the ex_vertexposition. So anything you do with the ex_vertexposition between these two lines is "vertex displacement." I have three examples in this post. The first manipulates the ex_vertexposition using sin(time) to create a blowing wind effect. The second uses a texture to displace the vertices. And the third uses the player position to influence the ex_vertexposition.

But since whatever you do with the ex_vertexposition between these two lines qualifies as vertex displacement, you can relegate it more to the realm of art than science. It's kind of just whatever "looks good." In the first vertex displacement example, if you go into the vertex stage of the shader and find the "wind" section there are these long, obnoxious lines. They yield a pretty convincing wind but they're not set in stone. I think a good rule is that it's only wrong if it looks wrong.

First, since we're talking about wind, I thought this would be a good place to drop a subsurface scattering shader. It's not true subsurface scattering, but it's a pretty good approximation. Subsurface scattering is that phenomenon that occurs when you put your hand over a flashlight and you can see the light coming through your hand. The light hits the one side of your hand and "scatters" under the surface of your skin, and comes out the other side as a glow.

Set the shrubbery in the scene and then in the Scene tab/General tab scale it to about 5x5x5. Then walk around it to view it from all sides. You'll see the effect:

33_subsurface scattering.zip

You may notice your shadows are blocky and not picking up the leaves. Use this shader to fix that by putting it in the "Shadow" slot in the material editor:

34_leaf shadows.zip

Here's the first vertex displacement shader. The effect occurs in time and I don't know how to make videos yet so I didn't bother making a picture. If you grab this shader you can just see it for yourself. Put the tree in the scene and scale it to something like 7x7x7. By the way, I made the tree and the shrubbery with TreeIt, which is a free program you can download somewhere. It's pretty cool and it makes the process of making trees and stuff much easier once you understand the program:

35_vertex displacement.zip

Next is a displace by texture shader. You use a simple heightmap in the vertex stage to warp the shape and it yields pretty cool results. A few points to note: vertex displacement warps the shape but it doesn't change the collision information. So don't expect to lay down a plane, slap a displacement shader on it and have an instant terrain (I wish!). Apparently shaders are handled on the GPU and collisions are handled on the CPU, so they don't mix. But you can still make cool looking stuff and there is a broad usefulness to it anywhere you want to add 3d effects. It's slower than parallax because you need the actual vertex count, but it doesn't "slip" the way parallax does and you can use it on complex shapes:

Another thing to note about vertex displacement is that the number of vertices and the space between vertices matters. If you have few vertices and they're far apart you're not going to get much of an effect. See the cube - which has only two triangles per face - compared to the sphere which has a lot of vertices close together:

36_displace by texture.zip

Another thing to note about this is that the normals don't change with the displacement. You can have a warped plane, but the light reacts to it as if it were still flat.  So in the above shader and here I added a section which creates a normal map from the heightmap. I then just mix the regular normals and the heightmap normals in the normals section of the fragment shader. You can change the resolution line but the range of about 200 - 500 appears to be the sweet spot:

37_normal from heightmap.zip

Lastly we have the interactive displacement shader, which is pretty cool if not rudimentary. You know how when you're playing a 3rd person game and you run through a bush the bush moves? This is a noob version of that. Set the shader on the shrubbery and then attach the script to the same shrubbery. Then drag the player into the slot. When you run through it moves a bit. You can adjust the interact power and the radius of the effect. No picture:

38_interactive displacement.zip

Happy shading!

Next: Depth Buffer...