Jump to content

Ultra Software Company Blog

  • entries
    189
  • comments
    1,264
  • views
    726,506

Contributors to this blog

Five Programming Changes You'll See in Leadwerks 5


Josh

6,741 views

 Share

Leadwerks 4.x will see a few more releases, each remaining backwards compatible with previous versions.  Leadwerks 5.0 is the point where we will introduce changes that break compatibility with previous versions.  The changes will support a faster design that relies more on multi-threading, updates to take advantage of C++11 features, and better support for non-English speaking users worldwide.  Changes are limited to code; all asset files including models, maps, shaders, textures, and materials will continue to be backwards-compatible.

Let's take a look at some of the new stuff.

Shared Pointers
C++11 shared pointers eliminate the need for manual reference counting.  Using the auto keyword will make it easier to update Leadwerks 4 code when version 5 arrives.  You can read more about the use of shared pointers in Leadwerks 5 here.

Unicode Support
To support the entire world's languages, Leadwerks 5 will make use of Unicode everywhere.  All std::string variables will be replaced by std::wstring.  Lua will be updated to the latest version 5.3.  This is not compatible with LuaJIT, but the main developer has left the LuaJIT project and it is time to move on.  Script execution time is not a bottleneck, Leadwerks 5 gains a much longer window of time for your game code to run, and I don't recommend people build complex VR games in Lua.  So I think it is time to update.

Elimination of Bound Globals
To assist with multithreaded programming, I am leaning towards a stateless design with all commands like World::GetCurrent() removed.  An entity needs to be explicitly told which world it belongs to upon creation, or it must be created as a child of another entity:

auto entity = Pivot::Create(world);

I am considering encapsulating all global variables into a GameEngine object:

class GameEngine
{
public:  
  std::map<std::string, std::weak_ptr<Asset> > loadedassets;
  shared_ptr<GraphicsEngine> graphicsengine;
  shared_ptr<PhysicsEngine> physicsengine;
  shared_ptr<NetworkEngine> networkengine;
  shared_ptr<SoundEngine> soundengine;
  shared_ptr<ScriptEngine> scriptengine;//replaces Interpreter class
};

A world would need the GameEngine object supplied upon creation:

auto gameengine = GameEngine::Create();
auto world = World::Create(gameengine);

When the GameEngine object goes out of scope, the entire game gets cleaned up and everything is deleted, leaving nothing in memory.

New SurfaceBuilder Class
To improve efficiency in Leadwerks 5, surfaces will no longer be stored in system memory, and surfaces cannot be modified once they are created.  If you need a modifiable surface, you can create a SurfaceBuilder object.

auto builder = SurfaceBuilder::Create(gameengine);
builder->AddVertex(0,0,0);
builder->AddVertex(0,0,1);
builder->AddVertex(0,1,1);
builder->AddTriangle(0,1,2);
auto surface = model->AddSurface(builder);

When a model is first loaded, before it is sent to the rendering thread for drawing, you can access the builder object that is loaded for each surface:

auto model = Model::Load("Models\box.mdl", Asset::Unique);
for (int i=0; i<model->CountSurfaces(); ++i)
{
	auto surface = model->GetSurface(i);
	shared_ptr<SurfaceBuilder> builder = surface->GetBuilder();
  	if (builder != nullptr)
	{
		for (int v=0; v < surface->builder->CountVertices(); ++v)
		{
			Vec3 v = builder->GetVertexPosition(v);
		}
	}
}

98% of the time there is no reason to keep vertex and triangle data in system memory.  For special cases, the SurfaceBuilder class does the job, and includes functions that were previously found in the Surface class like UpdateNormals().  This will prevent surfaces from being modified by the user when they are in use in the rendering thread.

A TextureBuilder class will be used internally when loading textures and will operate in a similar manner.  Pixel data will be retained in system memory until the first render.  These classes have the effect of keeping all OpenGL (or other graphics API) code contained inside the rendering thread, which leads to our next new feature...

Asynchronous Loading
Because surfaces and textures defer all GPU calls to the rendering thread, there is no reason we can't safely load these assets on another thread.  The LoadASync function will simply return true or false depending on whether the file was able to be opened:

bool result = Model::LoadASync("Models\box.mdl");

The result of the load will be given in an event:

while (gameengine->eventqueue->Peek())
{
	auto event = gameengine->eventqueue->Wait();
	if (event.id == Event::AsyncLoadResult)
	{
		if (event.extra->GetClass() == Object::ModelClass)
		{
			auto model = static_cast<Model>(event.source.get());
		}
	}
}

Thank goodness for shared pointers, or this would be very difficult to keep track of!

Asynchronous loading of maps is a little more complicated, but with proper encapsulation I think we can do it.  The script interpreter will get a mutex that is locked whenever a Lua script executes so scripts can be run from separate threads:

gameengine->scriptengine->execmutex->Lock();
//Do Lua stuff
gameengine->scriptengine->execmutex->Unlock();

This allows you to easily do things like make an animated loading screen.  The code for this would look something like below:

Map::LoadASync("Maps/start.map", world);
while (true)
{
	while (EventQueue::Peek())
	{
		auto event = EventQueue::Wait();
		if (event.id == Event::AsyncLoadResult)
		{
			break;
		}
	}
	
	loadsprite->Turn(0,0,1);
	world->Render();// Context::Sync() might be done automatically here, not sure yet...
}

All of this will look pretty familiar to you, but with the features of C++11 and the new design of Leadwerks 5 it opens up a lot of exciting possibilities.

  • Upvote 5
 Share

18 Comments


Recommended Comments

Another nice thing about shared pointers is we can safely return a Map object from Map::Load that contains a list of entities, and the user doesn't have to delete it.  You can use it or ignore it and let it go out of scope:

auto map = Map::Load("Maps/start.map",world);
for (int i=0; i<map->CountEntities(); i++)
{
	auto entity = map->GetEntity(i);
}
//nothing to delete or clean up!

 

Link to comment

With the creation of a GameEngine class, does this mean we could have two worlds loaded at once?

Say for example, have level 1 loaded and playing, then in the background, spawn a new GameEngine to load level 2 for an almost seamless transition?

Link to comment
2 minutes ago, martyj said:

With the creation of a GameEngine class, does this mean we could have two worlds loaded at once?

Say for example, have level 1 loaded and playing, then in the background, spawn a new GameEngine to load level 2 for an almost seamless transition?

I don't recommend creating multiple game engine objects because they could not exist in the same graphics context, but I believe you could create another world and safely load another map into that world in the background.  This is part of the reason the idea of a "current" world is going away.

  • Upvote 1
Link to comment

That async loading style seems to be a pain. Ideally I'm guessing there would be one place where the event loop exists, but loading of models may exist in another place. So how do you get the resulting model thst was loaded to the place that called the load async function?

C# handles this with an idea of async/await. It's a form of coroutine where you prefix an async function with await and it'll leave that function and continue with the main app loop and enter back in where it left off when that operation is finished. From the programmers point of view you write sequential code that's doing asynchronous stuff. Sadly C++ doesn't have this directly but it is proposed. However all that's really doing is a state machine behind the scenes to make it seem easy. It seems like C++ has some built in libraries that can mimic this behavior in c++11. It's called future library. I haven't used it but it may help get a more streamlined way of loading stuff async without needing some kind of event loop to go through which messes up the program flow and will probably result in bad designs and/or global variables required.

 

Link to comment
25 minutes ago, Rick said:

It seems like C++ has some built in libraries that can mimic this behavior in c++11. It's called future library. I haven't used it but it may help get a more streamlined way of loading stuff async without needing some kind of event loop to go through which messes up the program flow and will probably result in bad designs and/or global variables required.

 

I will look into this.  Another way is to create a separate thread and execute the loading command on that.

Maybe something like this would be better:

auto task = Model::AsyncLoad("Models\box.mdl");
while (!task->Completed())
{
	//Do some other stuff or return to the main loop
}
if (task->Result())
{
	auto model = static_cast<Model>(task->LoadedObject());
}
19 minutes ago, jen said:

Can you make Version 4.X available in the archive when version 5 arrives since those 2 versions are essentially different programs. I'm not really keen on changing my 10,000 code framework to fit the new formula.

Leadwerks 4.x will always be available on Steam.  The documentation will be moved to a "learn/4" folder on the site.  Leadwerks 5 will probably be a new app ID, if it is even released on Steam (it probably will but there's no telling yet where Steam will be by that point).

Keep in mind this is all a long ways off, but it's never too soon to start thinking about the future.  I definitely would not put any projects on hold for version 5, because it could be a long way away.

Link to comment

With the removal of world:getcurrent it would make scripts like my grenade script harder to make. I would need to know ahead of time, pre compile what world a prefab/entity will be loaded into, instead of asking what world spawned an entity. Also keep update,render,sync separate. A game server usually does not need to render or sync, but games where you might use buffers for game mechanics might need to renderer and not sync.

Link to comment
4 minutes ago, Einlander said:

With the removal of world:getcurrent it would make scripts like my grenade script harder to make. I would need to know ahead of time, pre compile what world a prefab/entity will be loaded into, instead of asking what world spawned an entity. 

I cannot find where your script is using the World::GetCurrent() commend.  You will still be able to retrieve the world an entity is created in.

Link to comment

A entity would need to explicitly need to be told what world to spawn into. Well when I load the prefab, to what world is it assigned to? It is not a child of any other entity. A drop in script would not have any clue what world it is being loaded into. So unless I can find what world I'm in then I would be stuck.

Link to comment
5 minutes ago, Einlander said:

A entity would need to explicitly need to be told what world to spawn into. Well when I load the prefab, to what world is it assigned to? It is not a child of any other entity. A drop in script would not have any clue what world it is being loaded into. So unless I can find what world I'm in then I would be stuck.

You would just do this:

function Script:Start()	
	self.GrenadePrefab = Prefab:Load(self.GrenadePrefabPath, self.entity.world)

And PickupWeapon.lua would pass it along like this:

function Script:Start()
	if self.vwepfile~="" then
		local prefab = Prefab:Load(self.vwepfile, self.entity.world)

I am not trying to make every command thread-safe, but eliminating bound states like this will make it simpler for me to program the internals of the engine, and it does allow some new multithreading techniques you can probably use.

:mellow:

  • Upvote 2
Link to comment

So what about bound-states like your GraphicsDriver and what not? Will you remove these? I feel like that'd be a slightly harder one to remove cause it's require lots of design changes, but on the flip side it may not be neccesary to remove it because you won't have the main thread writing to the GraphicsDriver at any point. (I don't think.)

I'm more asking, do you plan to remove ALL bound states?

Link to comment
3 hours ago, Crazycarpet said:

So what about bound-states like your GraphicsDriver and what not? Will you remove these? I feel like that'd be a slightly harder one to remove cause it's require lots of design changes, but on the flip side it may not be neccesary to remove it because you won't have the main thread writing to the GraphicsDriver at any point. (I don't think.)

I'm more asking, do you plan to remove ALL bound states?

Not sure yet.  If I use a gameengine object then it would contain the graphicsengine (graphicsdriver) as a member.  In multithreaded programming this makes life a lot easier, but I don't want to pedantically enforce something just because.

Link to comment

Thing that worries me about this subject is I don't have a clue what any of you are talking about. Given a choice I will just use old fashioned c language. Its easy to learn, its fast and could do the job.

Link to comment
1 hour ago, cassius said:

Thing that worries me about this subject is I don't have a clue what any of you are talking about. Given a choice I will just use old fashioned c language. Its easy to learn, its fast and could do the job.

It's basically this without delete.

Link to comment
20 hours ago, Josh said:

Not sure yet.  If I use a gameengine object then it would contain the graphicsengine (graphicsdriver) as a member.  In multithreaded programming this makes life a lot easier, but I don't want to pedantically enforce something just because.

I get why you'd want to do this but would this not be the same thing in principal because your "GameEngine" object would nwo be the bound state? This would have the same implications n a multithreaded environment as GetCurrent() because like the latter, only one thread could write to this "GameEngine" state at a time. Reading is always thread safe.

I guess it doesn't matter because users don't have to play with that stuff, I just figure why not make life easier for yourself? :P

Link to comment
11 hours ago, jen said:

I would love to see support for double precision.

Enterprise edition may have this, both for physics and optional graphics, together with a floating-point depth buffer.

Link to comment
50 minutes ago, jen said:

I'm curious, is there a point carrying on with Leadwerks 4.5/6 development if you're going to recreate the application in version 5 anyway?

Josh has an existing customer group that is using the engine for their (commercial) games. He can't simply say "alright I am going to put all my time in to this new program, that might be out in 1/2/3 years from now.".  Some time will go in maintaining the current engine and some time will go in to planning/making the new engine/editor.

Link to comment
5 hours ago, jen said:

I'm curious, is there a point carrying on with Leadwerks 4.5/6 development if you're going to recreate the application in version 5 anyway?

Everything new will carry over to version 5.  The only thing I am reluctant to put a lot of time into is additional features in the editor, since that code is eventually going to be replaced.

Link to comment
Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...