Single-state Lua

[Josh]

I can see already Lua is being very well-received, and it looks like some serious work is going to be done with it. I did not plan on making any revision to the system, but if we are going to make any changes, it should be done now, not later.

 

I do know a way I can use a single state and still have our same function name conventions. Internally, the engine can just do this string before loading a new class script:

Spawn=nil
InitDialog=nil
Update=nil
ReceiveMessage=nil

 

Then let's say light_directional.lua is run. After that, this code can be run silently by the engine:

light_directional_Spawn=Spawn
light_directional_InitDialog=InitDialog
light_directional_Update=Update
light_directional_ReceiveMessage=ReceiveMessage

 

The engine would then call the functions prefixed with the model name, like light_directional_Update() instead of Update(). You could still call the functions the same thing in each class script, but the engine would be able to differentiate between them.

 

Consequences:

-You can access lua functions across scripts.

-You can't call Update(), etc. yourself because the functions are internally renamed.

-You have to clean up everything in the Cleanup() function since the state won't be deleted when all instances are cleared.

-The Lua virtual machine will get bigger and bigger as more model classes are loaded. Maybe I can set all the class functions to nil when the class is deleted.

 

I initially thought Lua would be good for tinkering around with and making some cute demos, but it looks like people like it and are very capable of using it. A single-state system might be better in the long run, and it's not that big of a revision.

[Rick]

I'm all for this, but is it really that simple? Won't you run into other issues with how lua scripts are currently used? entitytable for example will now be global and all entities loaded will be in it not just the entities local to one class model. Plus all scripts currently need to dofile() to base to use constants and stuff. That won't be needed because it'll only needed to be called once to load all that stuff.

 

Like I said I'm all for it but I would assume there are more changes than what you stated. Maybe have some of us test out the new way and see how things work out.

[TylerH]

I don't want this to happen. My vote is no.

[Josh]

I'm all for this, but is it really that simple? Won't you run into other issues with how lua scripts are currently used? entitytable for example will now be global and all entities loaded will be in it not just the entities local to one class model. Plus all scripts currently need to dofile() to base to use constants and stuff. That won't be needed because it'll only needed to be called once to load all that stuff.

The base file can be called once by the main program, or can be placed in the start folder. The entity table would be global, but it's easy to iterate through all instances of a model. Undoubtedly some small changes will have to be made to the model scripts, but we're still in the stage where that is acceptable.

 

I don't want this to happen. My vote is no.

Can you elaborate why?

 

The biggest disadvantage of the multistate system is when you have tables that have to be shared, it is impossible. We might be able to get around that for a long time, but I have a feeling it is going to come back and bite us in the future. For example, if I want to tell if a vehicle's wheel is on the road, I would want to be able to call a function from all nearby road entities that would be written in the road script. That's where it gets ugly, and I am afraid we will run into more cases like that as we make more complicated games.

 

I won't do anything to the official release until I am sure this is a good idea. If I pursue this, it will start by posting some test versions of the interpreter so we can play around with the idea.

 

Like I said, I don't like making changes, but if we are going to, it should be done now while it won't hurt anything.

[TylerH]

Eh, now that you explained that it won't be mandatory until it works, that is good.

[Nilium]

My vote is for yes, and it should count for 10 votes since I wrote LuGI.

[TylerH]

Uhm, the fact you wrote a Lua wrapper for Blitzmax doesn't mean your vote should count for more, nor did I find it humorous if it was supposed to be as such. You need to prove your expertise on the variable state field.

 

List some pros/cons of both multi-state and single-state in your own opinion, if you may.

 

 

Also, since you coded the wrapper I believe Josh uses, are you the one I need to bug to be able to pass Lua tables and numbers to and from Blitzmax?

[Josh]

Nilium does know a lot more about Lua than I do.

 

I'm not sure if Lua can pass a pointer to a table back to the main program. Everything I have searched for and read indicates otherwise. Maybe there is a way, but I couldn't figure it out. In the end, the way I ended up associating a table with a BMX object was to put them both into a Lua table. That's why every function starts with a lookup in the entitytable to retrieve the table associated with a model.

[TylerH]

Well, if we had some way of having SendMessage and ReceiveMessage be entirely Lua based, we can pass Lua tables to and from them.

 

I just need a system that works for in Lua, and only for Lua. The fact SendMessage and ReceiveMessage have to work with the C++ and BMX portions as well is what causes the problem.

 

I think logic entities will fix most of this issue.

[Rick]

Well, if we had some way of having SendMessage and ReceiveMessage be entirely Lua based, we can pass Lua tables to and from them

 

I think this would be a nightmare. I don't see a point in sending entity messages to move variables around when the variable should just be global.

[TylerH]

Sure, because it is ultra viable to have global variables for every bullet that has loads of hit info to pass to its victim entity.

 

That would be the nightmare if you were using anything more than a pistol and had more than ~10 bullets on screen.

 

[/sarcasm]

 

 

I am talking about one entity passing a table of information to another entity. For things like a single player's data in a singleplayer game, this could definitely be a global object. But in the case of say discreet information that is easier to pass in a table form from one entity to another in a single message is much more viable.

 

For example, I will eventually have this:

bullet -> SendMessage containing "damage amount" to -> prop

bullet -> SendMessage containing "damage type" to -> prop

bullet -> SendMessage containing "damage force" to -> prop

bullet -> SendMessage containing "attacker" to -> prop

bullet -> SendMessage containing "etc." to -> prop

...

You get the point?

 

It is much easier to do:

bullet -> SendMessage containing hit info table (damage, amount, type, force, attacker, weapon, etc.) to -> prop

 

And in that case, global variables also would fail, since you would now need to be prefixing of suffixing your variable names with some ID per bullet, which is ridiculous.

[Nilium]

Uhm, the fact you wrote a Lua wrapper for Blitzmax doesn't mean your vote should count for more, nor did I find it humorous if it was supposed to be as such. You need to prove your expertise on the variable state field.

 

List some pros/cons of both multi-state and single-state in your own opinion, if you may.

 

 

Also, since you coded the wrapper I believe Josh uses, are you the one I need to bug to be able to pass Lua tables and numbers to and from Blitzmax?

 

I was being fairly serious about my opinion being more valuable than that of the unwashed masses here. I wrote the code that allows you to do what you can (edit: to clarify, I mean LuGI, not the engine/editor code using Lua, since I have my disagreements about the way it looks to be implemented [which Josh has his own reasons for]), and chances are I know a fair bit more about what Josh is describing than the majority of the people here. As such, my opinion has more value than many of the others'.

 

Pros of multi-state:

• Everything is definitely separate
  
• The different states can run in parallel
  

Cons:

• Memory use sky-rockets
  
• You're going to be running as many garbage collectors as you have states (this is expensive)
  
• Communication between states is incredibly difficult
  
• Accordingly, everything you share is duplicated between states, and keeping those objects in sync can be a nightmare
  

 

Pros of single-state:

• You can have separate execution contexts, as with the multi-state setup, by using Lua threads
  
• Sharing data between threads is easy, keeping things in sync is easy.
  
• One garbage collector handling it all, less expensive overall than 30 to 100 or more garbage collectors running every X frames.
  
• Memory consumption is significantly less.
  

Cons:

• You can't entirely guarantee there's zero communication between threads/execution contexts (it's not really an issue, but I suppose in a secure environment, such as in a server, you would want them 100% separate).
  
• Lua threads do not run in parallel
  

 

As far as I'm concerned, there's no good reason to be using multi-state unless you're running a web server that uses PayPal or some other payment stuff in Lua, at which point I would question your sanity for attempting that.

 

Now, you can pass Lua tables to BlitzMax as an array (or a TMap, I suppose), and you can pass arrays to Lua as tables currently. However, I am phasing this out with support for direct array access, since converting to/from arrays and tables is a speed and memory hit that doesn't need to occur. Ideally, this shouldn't affect any scripts unless you were sort of abusing tables created from arrays, at which point you'll have to implement a new routine for converting arrays to tables (or just copy the routine in the master branch of LuGI). If there's some other sort of problem that you think I'm supposed to address, you should be less vague in stating what problem it is.

 

Following that, I'm not sure what problem with passing numeric values to/from Lua you're having, but I can assure you that if you have trouble passing numbers to/from BlitzMax/Lua, you do not have a whole lot of room for talking.

 

I'm not sure if Lua can pass a pointer to a table back to the main program.

You can pass a pointer to a table back to the program, but this is a one-way thing. You cannot take the pointer and put it back on the stack. I'm not sure what the reason is, but the function for getting the pointer (lua_topointer) is mainly for debugging purposes. There's probably a good reason, I'm sure if you asked on a Lua mailing list they'd have some answers. This limitation has never actually affected me, so I didn't spend any amount of time looking it up until you'd asked about it some time ago.

[Josh]

Pros of multi-state:

• Everything is definitely separate

This is the biggest advantage.

• The different states can run in parallel

This is not a concern, since no game engine can call random commands on separate threads without locking everything.

 

Cons:

• Memory use sky-rockets

Is memory use by Lua even an issue?

• You're going to be running as many garbage collectors as you have states (this is expensive)

Is it more expensive to run multiple GCs with a small amount of data, or one GC with a lot of data? I have not seen any definitive data on this.

• Communication between states is incredibly difficult

I've demonstrated that almost every interaction can be done, but it is awkward, and I do think it will cause problems further down the road. This is the single best argument for a single state.

• Accordingly, everything you share is duplicated between states, and keeping those objects in sync can be a nightmare

I don't think this is a big issue unless you are trying to store complex sets of data. I could see it coming up during advanced AI implementations.

 

Pros of single-state:

• You can have separate execution contexts, as with the multi-state setup, by using Lua threads

Can't comment on this.

• Sharing data between threads is easy, keeping things in sync is easy.

True.

 

• One garbage collector handling it all, less expensive overall than 30 to 100 or more garbage collectors running every X frames.

Are you sure? Where is the proof of this? I have seen the opposite asserted.

 

• Memory consumption is significantly less.

Is memory consumption of Lua even an issue?

 

Cons:

• You can't entirely guarantee there's zero communication between threads/execution contexts (it's not really an issue, but I suppose in a secure environment, such as in a server, you would want them 100% separate).

The biggest con is duplicate function names and garbage collection. Right now cleanup is simple because when all instances of a model are freed, the lua state for that reference is freed. Cleanup becomes more important when a single-state is used, especially as individuals scripts are saved and re-run by the state in the editor. (When you are real-time coding a model).

 

• Lua threads do not run in parallel

Not a concern.

 

 

So the big issues are memory leak prevention, especially when a script is being re-run over and over as it is edited and saved, and sharing data between scripts.

[TylerH]

I have quite a bit of room to talk

 

According to Josh, the Lua implementation he uses doesn't allow integers to be passed to BlitzMax and then back to Lua via his Entity Message System, thus I have to send them as strings and get them back out as strings using tostring and tonumber respectively.

 

Sending direct integers nags me with Lua errors: "The object at index(3) is not a valid object/value". That occurs when I try to pass 0 as an integer to the 2nd parameter of SendMessage in Lua. But if I pass 0 as "0", no issues what-so-ever.

 

I am not playing the blame game, I just want whoever has the power to fix this to fix it.

[Nilium]

Are you sure? Where is the proof of this? I have seen the opposite asserted.

It's mainly conjecture. To me, 100 garbage collectors means you're calling on a different garbage collector 100 times. If you can do it all in one shot, I would say it's less expensive than collecting the same object 100 times over multiple GCs (e.g., share an object between all the states, each GC has to work with that object, as opposed to having one GC that collects one object [LuGI will only ever push one instance of an object to a given state, so duplicates don't occur]). Furthermore, I doubt creating and closing states frequently is going to be very fast. Any potential speed benefits gained by using multiple garbage collectors probably won't be applicable in a game engine. Again, perhaps something like a server would be better-suited to using multiple states.

 

Is memory use by Lua even an issue?

It is as far as I'm concerned. He asked for my opinion, after all, and I gave mine - if memory use isn't a concern of yours, then that's fine.

 

This is not a concern, since no game engine can call random commands on separate threads without locking everything.

That's not entirely true, but really it's unrelated to Leadwerks.

 

According to Josh, the Lua implementation he uses doesn't allow integers to be passed to BlitzMax and then back to Lua via his Entity Message System, thus I have to send them as strings and get them back out as strings using tostring and tonumber respectively.

This is a flaw due to his use of multiple states and probably not writing the function properly. It's entirely avoidable if you just check the type at the index, but he probably wrote that routine and wrapped it using LuGI, which tries to match the BlitzMax method as exactly as it can. As such, this isn't an issue I need to address.

[Josh]

SendMessage() is an engine command, so it handles data native to the engine, which doesn't include Lua tables.

 

It is as far as I'm concerned. He asked for my opinion, after all, and I gave mine - if memory use isn't a concern of yours, then that's fine.

I mean, does Lua consume an amount of memory significant enough to worry about?

 

It seems we want some data to be shared, but we don't want other data to be. Entities should be able to communicate and call each others' special functions, but we also want model scripts to be simple to write. There are two ways this can be solved with only small modifications:

 

1. Rename all model script functions with a prefix of the model file name. Instead of Update() you would write light_directional_Update().

 

2. Make the engine automatically rename these function names by running a short source code after the script file is run:

light_directional_Update=Update
Update=nil

 

I am leaning towards option 1. It's more explicit and there are no issues of trying to call a special function later only to find it has been set to nil automatically. The entire combined source code of the state can be printed out, and it is clear where everything occurs. I don't like ambiguity as a general thing. Option 1 is not as cool as just writing "Update()" in each script, but it seems like the most explicit and understandable approach.

[TylerH]

I said INTEGERS, which are NOT Lua Tables. It crashes with Integer variables...

[Josh]

The function calls for an object, so an integer would not work. Lua is a little different because integers, strings, objects, and tables can all be used in the same parameter. This function has to work with C++ and every other language, so the flexibility of Lua is not something it can support.

[TylerH]

I see, and like I said it isn't like I haven't overcome the problem already. I just am saying that it is strange that it will take strings and not integers, booleans, though I guess it is obvious since the strings are objects and strange that the integers are not.

 

It should be able to somehow cast to raw userdata and back, though I can't say that is the least memory intensive.

[Josh]

When I think about the single state with renamed entity functions, it makes sense when I consider the resulting combined source with the main program, and all the included model scripts. The source code the engine makes by doing different script files would look something like this:

 

Graphics(800,600)

while KeyHit()==0 do
update()
render()
flip()
end

function light_directional_Spawn(model)
--do some stuff
end

function light_directional_Kill(model)
--do stuff
end

function vehicle_truck_Spawn(model)
--do some stuff
end

function weapon_gun_Spawn(model)
--do stuff
end

 

It looks a lot like a C program, like the Quake 3 source for example. It makes sense and seems like a good approach.

[TylerH]

Until you read the PIL (http://apache.dataloss.nl/~peter/www.lua.org/pil/) Book, and learn about true OO and Inheritence methods in Lua via metatables and the like.

 

If I can grasp it, I am sure you can use the techniques presented there to innovate a nice approach.

[Josh]

Well, I started reading, and even if I can make sense of it, if it isn't immediately obvious my users won't get it. And the whole point of Lua is to be easy.

 

With a single-state system, here is what the directional light script looks like:

require("scripts/base")

--Initially set all the functions we might want, so we
--don't have to keep track of which ones we actually declare.
light_directional_InitDialog=base_InitDialog
light_directional_Spawn=base_Spawn
light_directional_GetKey=GetKey
light_directional_SetKey=SetKey
light_directional_ReceiveMessage=base_ReceiveMessage
light_directional_Kill=base_Kill

function light_directional_InitDialog(grid)
base_InitDialog(grid)
group=grid:AddGroup("Light")
group:AddProperty("Resolution",PROPERTY_CHOICE,"256,512,1024,2048")
group:AddProperty("linearoffset",PROPERTY_VEC3,"0,1,2","Linear offset" )
group:AddProperty("shadowdistance",PROPERTY_VEC3,"","Shadow distance" )
group:AddProperty("Range",PROPERTY_FLOAT)
group:Expand(1)
end

function light_directional_Spawn(model)
local entity=base_Spawn(model)
entity.model:SetKey("resolution","2")
entity.light=CreateDirectionalLight(entity.model)
return entity
end

function light_directional_Kill(model)
local entity=entitytable[model]
if entity~=nil then
	if entity.light~=nil then
		entity.light:Free()
		entity.light=nil
	end
end
base_Kill(model)
end

function light_directional_GetKey(model,key,value)
local entity=entitytable[model]
if entity==nil then return value end
if entity.model==nil then return end
if entity.light==nil then
	return base_GetKey(model,key,value)
else
	if key=="linearoffset" then
		return entity.light:GetShadowOffset(0,0)..","..entity.light:GetShadowOffset(0,1)..","..entity.light:GetShadowOffset(0,2)
	elseif key=="shadowdistance" then
		return entity.light:GetShadowDistance(0)..","..entity.light:GetShadowDistance(1)..","..entity.light:GetShadowDistance(2)
	elseif key=="range" then
		return entity.light:GetRange()
	elseif key=="shadowresolution" then
		resolution=entity.light:GetShadowmapSize()
		if resolution==256 then
			return 0
		elseif resolution==512 then
			return 1
		elseif resolution==1024 then
			return 2	
		elseif resolution==2048 then
			return 3
		else
			return -1
		end
	else
		return base_GetKey(model,key,value)
	end
end
end

function light_directional_SetKey(model,key,value)
local entity=entitytable[model]
if entity==nil then return 1 end
if entity.model==nil then return 1 end
if entity.light==nil then
	return base_SetKey(model,key,value)
else
	if key=="resolution" then
		if value=="0" then
			entity.light:SetShadowmapSize(256)
		elseif value=="1" then
			entity.light:SetShadowmapSize(512)
		elseif value=="2" then
			entity.light:SetShadowmapSize(1024)
		elseif value=="3" then
			entity.light:SetShadowmapSize(2048)
		end
	elseif key=="range" then
		entity.light:SetRange(value)
	elseif key=="shadowdistance" then
		local offset=string.Explode(value,",")
		x=tonumber(offset[1])
		y=tonumber(offset[2])
		z=tonumber(offset[3])
		if x==nil then x=0 end
		if y==nil then y=0 end
		if z==nil then z=0 end
		entity.light:SetShadowDistance(x,0)
		entity.light:SetShadowDistance(y,1)
		entity.light:SetShadowDistance(z,2)	
	elseif key=="linearoffset" then
		local offset=string.Explode(value,",")
		x=tonumber(offset[1])
		y=tonumber(offset[2])
		z=tonumber(offset[3])
		if x==nil then x=0 end
		if y==nil then y=0 end
		if z==nil then z=0 end
		entity.light:SetShadowOffset(x,1.0,0)
		entity.light:SetShadowOffset(y,1.0,1)
		entity.light:SetShadowOffset(z,1.0,2)
	else
		return base_SetKey(model,key,value)
	end
end
return 1
end

[Rick]

light_directional_InitDialog=base_InitDialog

light_directional_Spawn=base_Spawn

light_directional_GetKey=GetKey

light_directional_SetKey=SetKey

light_directional_ReceiveMessage=base_ReceiveMessage

light_directional_Kill=base_Kill

 

That looks to be about the only new thing. You could probably encapsulate that with a function call.

 

Maybe like:

 

InitEntity("light_directional")

 

 

You should be able to do:

function InitEntity(ent)

_G[ent.."_InitDialog"] = base_InitDialog

...

end

[TylerH]

Rick has the right idea.

 

If we were to go to single-state, I would have to ask for it to be in that form.

 

I don't want to have to start prefixing my functions in code, the engine, if it wants them that way, will have to do that itself.

[TylerH]

I still think you are trying to reinvent the wheel here as well...

 

Lua has such great support for packages, tables, prototyping, etc., all you have to do is some work for it to be efficient and easy to use.