Lua Garbage Collection

[Haydenmango]

I have been reading a lot about Lua and specifically the garbage collector recently and learned a few things. Such as -

The garbage-collector pause controls how long the collector waits before starting a new cycle. Larger values make the collector less aggressive. Values smaller than 100 mean the collector will not wait to start a new cycle. A value of 200 means that the collector waits for the total memory in use to double before starting a new cycle.

The step multiplier controls the relative speed of the collector relative to memory allocation. Larger values make the collector more aggressive but also increase the size of each incremental step. Values smaller than 100 make the collector too slow and can result in the collector never finishing a cycle. The default, 200, means that the collector runs at "twice" the speed of memory allocation.

source- http://www.lua.org/manual/5.1/manual.html

 

I have been testing my game for memory issues for weeks now and I am starting to think that the garbage collector just isn't fast enough to keep up with the amount of things I have going on. I noticed that since my memory usage rises to quick that the collector waits for my total memory to double before starting the next cycle and things start to get out of hand from there.

I know that in most cases the garbage collector should be able to handle by itself but in my case it is not. I do not have any memory leaks in my code. I know this because if I run the garbage collector every frame my memory usage stays consistent.

 

So could it be a better option to control the Lua Garbage Collector manually in my case?

 

Edit ---- I have tested using collectgarbage('setpause',100) in my App Start() function.

My game lasted about 2 hours longer than usual!! I normally test my games memory usage by going into release mode and standing completely still until my game crashes(It used to crash after about an hour); this time, well after I reached my normal 'crash point' I started running around doing crazy things to see if I would run out of memory but my game crashed due to an unknown error (I wasn't in debug mode) which I believe to be an issue in my script(not memory related)!!!

I found multiple bugs I haven't noticed in my game before as well which I will fix and then I will test this more thoroughly.

Also when tested in debug mode I noticed that my memory usage stayed consistent when before it used to constantly increase!

[YouGroove]

Edit ---- I have tested using collectgarbage('setpause',100) in my App Start() function.

My game lasted about 2 hours longer than usual!! I normally test my games memory usage by going into release mode and standing completely still until my game crashes(It used to crash after about an hour); this time, well after I reached my normal 'crash point' I started running around doing crazy things to see if I would run out of memory but my game crashed due to an unknown error (I wasn't in debug mode) which I believe to be an issue in my script(not memory related)!!!

I found multiple bugs I haven't noticed in my game before as well which I will fix and then I will test this more thoroughly.

Also when tested in debug mode I noticed that the my memory usage stayed consistent when before it used to constantly increase!

 

To resume this seems mainly problems in your code , and LE3 just runs fine.

We should not have to call garbage collector, when you will have your code fixed, you should not need to call it.

[Haydenmango]

That is not what I am saying at all. What I am saying is that the Lua Garbage Collector is sometimes not fast enough for games that are using lots of memory so this function helps if you are creating games with big worlds.

 

I found multiple bugs I haven't noticed in my game before as well which I will fix and then I will test this more thoroughly.

The bugs I found were simple issues that weren't causing memory leaks. My game runs exactly the same now that I have fixed them and tested it again with and without collectgarbage('setpause",100).

If you must know the bugs were simple things like bees not doing damage because a condition wasn't see to true as well as my respawntimes for certain creatures were way to quick.

 

 

Oh and --

but my game crashed due to an unknown error (I wasn't in debug mode) which I believe to be an issue in my script(not memory related)!!!

The issue that crashed the game was a rare condition that only would happen if the bees were not doing damage(it was a physics issue the bees normally turn their physics off and start following the object while doing damage but the physics never turned off).

 

.... think about it; does one general setting for a garbage collection system work for every single program ever created?

[Rick]

So before that line of code when you would stand still and do nothing the game would crash? That seems very suspicious and I would really think it's the code you have vs anything Leadwerks is doing. Are you able to comment out chunks of code that provide some feature and test that way to narrow it down? Like, maybe comment out wolves, or birds and test?

 

Before that line was it crashing because it ran out of memory? How much memory does your system have?

[Haydenmango]

Are you able to comment out chunks of code that provide some feature and test that way to narrow it down? Like, maybe comment out wolves, or birds and test?

That's what I have been doing and I can't narrow it down to anything other than I have just to many things happening. I have done memory tests on everything for hours everyday on every line of every script and haven't added anything but simple bug fixes to my game for the past 2-3 weeks now. There aren't any problems I can find at this point.

 

So before that line of code when you would stand still and do nothing the game would crash? That seems very suspicious and I would really think it's the code you have vs anything Leadwerks is doing.

 

It is a powerful line of code if you haven't read into what it does. Also I am not blaming Leadwerks for doing anything I was asking if it is possible that the Lua Garbage Collector isn't fast enough for some situations.

My entities have no physics being applied to them, don't animate, don't move, aren't visible, and their scripts return end unless they are within a certain range so everything is as optimized as I can get it to be. When my entities are nearby some of them have sightly complex scripts that they use so even if I stand still the environment still generates decent amounts of memory.

 

Yes it would crash because of Lua Error:Out of Memory

6 GB RAM (5.89GB usable) on my system.

[Rick]

Have you released your game with source yet for others to check out and test with you? I don't think anyone so far thinks it's a Lua problem in general, but without seeing your code it's hard to say what could be the issue. I know you are confident in your ability to debug but others are confident in theirs as well and may think you are just missing something. Without seeing the code it's hard for us to say.

 

Or maybe you're holding out for Josh to test your code?

[Haydenmango]

I haven't and I am getting to the point where I may begin giving out my source code since I do agree with you on the point that I may be completely wrong about everything/some of the things I've done; after all I haven't been doing this for to long. Also I have like 15 or so scripts and at least 4 of them are 600-1000 lines long so it wouldn't really help to just post snippets in this case but again I will think about giving out the source code if I give up. I need to do more tests though still since I just started using this function and now each test run is 3+ hours.

 

No I know he is busy. If he wanted to adopt the game for the workshop or something that would be cool but I am not holding my breath.

[YouGroove]

Memory increase and going over 4Go for a non AAA complex next gen game indeed can be problematic if that can happen on indie open games outdoors running mulitple scripts and entities.

Perhaps it's how you manage entities and their attributes that can make problems , why not ?

Perhaps all scripts of 1000 lines is too much and you should instead make separate scripts than import them ?

 

If you could zip and post your project with just necessary assets and files to test, some of us could try and see how it goes.

[Josh]

The Lua garbage collector can't be "too slow". If your memory doubles, the GC will cut it back down, and nothing else can happen before the GC is finished. So it's always on top of things.

 

It's either a mem leak in your program, or possibly one in Leadwerks itself. Commenting out functionality is a good approach. If you call collectgarbage() each frame, leaks are usually pretty obvious.

[Haydenmango]

If your memory doubles, the GC will cut it back down, and nothing else can happen before the GC is finished.

This is something I have been testing a lot recently. Once my memory doubles the GC will cut it down but then it waits till my memory doubles the amount that it was at when it was cut down. Meaning if it collects memory at something like 800 then it will do its next memory collect at 1600 and the next at 3200 and then 6400 and then etc. Obviously the number gets to high eventually and then the game crashes due to being out of memory. This is what I have observed and is why I looked into this at all.

 

If I call collectgarbage() every frame there are no leaks my memory stays the same(without using collectgarbage('setpause',100) in my App Start() function). I wonder, why are there even controls for the GC if it supposedly runs perfectly on its own?

 

If you could zip and post your project with just necessary assets and files to test, some of us could try and see how it goes.

I haven't given up just yet. This project is a learning experience for me so I would rather not just give out my project and have all the answers given to me by other people.

 

Also I tested my game (with collectgarbage('setpause',100) in my App Start() function) all night without a crash or any other weird behaviour. It ran for 10 hours(8-9 hours longer than usual), I exited out because I wanted to read this forum post. I saw no bugs the whole time I was playing and everything was working great.