Hidden entities still get iterated over in LE?

[Rick]

If an entity is hidden does it somewhere get iterated over behind the scenes in LE still? It seems like it does but I wouldn't think it should. Also if it's not in view range it must still get iterated over.

[darthlukan]

It would make sense that an entity is iterated over even if it's hidden if the "hidden" attribute is set per-entity. The check would still have to be run and the only way to do that would be to iterate over all entities and see if "hidden" is "true", at which point it wouldn't run the render.

[Einlander]

It could have to lists, 1 render list and 1 ignore list. This way you would never have to iterate over the hidden list if you take it out the render list. This is how I would do it, but it's up to Josh to tell us how. View range determination / culling would require all entities that are not explicitly set to hidden to be iterated, else how would it know to render it?

[darthlukan]

That's a good suggestion, but ultimately, you'd still have to iterate over each entity to determine which list it belongs in. The difference now you've turned an O(n) problem into an O(n^2) problem since you're now iterating over two lists at some point.

[Einlander]

True, but you only have to push/pop when explicitly told to. If an object is hidden by code, and never shown again, it is never iterated. and you have a shorted list of things to iterate.

[Rick]

View range determination / culling would require all entities that are not explicitly set to hidden to be iterated, else how would it know to render it?

 

Couldn't you break the world into a logical grid? Each "tile" in this grid is a certain width/depth. When entities are added to the scene they are figured out what tile they would be on and added to a list inside that tile that stores entities in that tile. Finding out what tile is easy an division operation.

 

Then every frame or every x ms you find the tile the camera is on, again easy arithmetic, and then based on that tile row/col you can loop over ONLY tiles that surround the camera and ONLY over entities in those tiles. You save that list off each time and each new time you compare the 2 lists to see what is on both and what isn't to tell you what changed and what you need to hide/show for that iteration.

 

Now this wouldn't really allow each entity to define the view range but more of a view range around the camera/player, but it seems like it would be far more efficient when you end up with a ton of entities in the scene that aren't visible because of view range. To make it work on a per entity basis maybe infinite view ranges are left out and always iterated over (which should be far less or none) and the tile radius to check is the max view range and then as we iterate over the entities in these tiles we check their view range to see where they would fall and if they should be visible or not.

 

Again, all of this would make things more efficient when there are a lot of entities. I feel like right now the easy way of iterating over the entities was taken instead of possible more efficient ways.

 

More efficient ways for view range would be to do these things on another thread over multiple frames. With view range stuff I think it's fine if it's not instantly shown when entering view range. A few ms late seems perfectly acceptable I would think.

 

Entities with scripts attached and not hidden would need to be iterated over each frame but again this will be much fewer than just scenery in general.

 

So the point here is that there are more efficient ways to iterate over the entities in the map than just iterating over all of them on the main thread all the time.

 

 

Imagine the grid system I talk about. If each grid is 100mx100m and you have a 5x5 grid around the camera tile you are iterating 25 times and then sub-iterating over the lists in each tile. That could be a TON less iterations each time you need to do it.

 

Moving items change the tile lists they are on when they switch tiles. The player would also only need to iterate over this 5x5 grid IF they move tiles. So if your main tiles are fairly large that wouldn't be each frame but only when they move from tile to tile. Checking what tile they were on last frame vs this frame is a much cheaper operation than iterating over all entities each time. There just seems to be far better ways to handle this and I feel like what is in there is probably just the cheap and easy coding way of things which introduce these entity count limitations to LE that aren't needed.

[Josh]

A hidden entity removes itself from the scene octree.

[Rick]

So I'm curious why when I load 88,000 entities and hide them all, why I get 0 FPS.

 

156 polygons on screen, 0 shadow polygons, 1 light, 1 shadow rendered and 86MB used and I get 0 FPS.

 

The less entiites loaded (and still hidden) the higher the FPS. What else besides iterating over more entities could cause this slowdown?

[Josh]

What kind of entities are they? Do they have a script? Do they have physics?

 

If I add this at the beginner of main.lua it has no effect on the speed:

for n=0,100000 do
local m = Model:Create()
m:Hide()
end

[Rick]

They are the tree entities that come with LE. No scripts but prefabs with view range set. Not sure on physics, it's whatever the tree model has on it. Guessing it does, but not sure why that would matter since it's hidden and shouldn't be iterated over right?

[Rick]

When I do the below in main.lua I get 16fps for about a second before it crashes. 100k didn't even show me the text of FPS it just froze.

 

local m = Model:Load("Models/Trees/pine01.mdl")

for n=0,20000 do
local n = m:Instance()
n:Hide()
end

 

I also did it the below and same thing:

 

for n=0,20000 do
local m = Model:Load("Models/Trees/pine01.mdl")
m:Hide()
end

 

Both methods should be the same, making instances, but wanted to double check with both approaches.

 

Your example with just Model:Create() works just fine, but clearly something else is going on with other models. Even those that come with LE by default.

[Slastraf]

When I hide "remove" a crawler in my game crawler defense , the turret seems to still focus on it. The turret has a target and alligns to the enemy set in a scene. When I hide the crawler and it is nowhere shown again in the script, it is just invinsible (for me).

I need to think of an other way for when a crawler gets trought the whole path in the tower defense game.

[Einlander]

When I hide "remove" a crawler in my game crawler defense , the turret seems to still focus on it. The turret has a target and alligns to the enemy set in a scene. When I hide the crawler and it is nowhere shown again in the script, it is just invinsible (for me).

I need to think of an other way for when a crawler gets trought the whole path in the tower defense game.

 

Well of course, the entity still exists. So your script will still track it. You would to add a hidden entity check to tell the turret to not aim at it.

[Josh]

This is probably due to the object being in the physics system, since the pine tree has a shape file. 100,000 objects crashing the physics system or causing slowdown isn't a bug.

[Rick]

Can't suspend from the physics system? I mean it's hidden it shouldn't be looped over in any system then right? Just seems a little dismissive seeing as this is something that hinders maps with a large amount of entities in it even when they aren't visible from being possible.

[darthlukan]

Is there a way to force removal of the hidden entities from the physics system? For example, let's say I hide an entity, can I then do something like: "model:RemovePhysics()" so that item is cleared from the calculations? I'm assuming you have a way to remove items from the physics calculations as a private method, would exposing it as an API method cause unsafe execution? For large maps with large amounts of entities, it might be worthwhile to expose something like that so that developers can manage resources with a bit more control instead of relying on the Engine to automatically handle it.

 

Just a thought.

[Einlander]

Can someone test this for me since I'm at work?

 

Set the collision type to none

http://www.leadwerks.com/werkspace/page/documentation/_/command-reference/collision-r778

 

And then set the mass to 0

 

And check performance. It may help, may not.

[Josh]

If you set the shape to NULL and mass to zero it will likely remove the entity from the physics simulator.

 

What's the point of this, anyways? What are you doing that requires this?

[Rick]

I guess the point is that this affects all of our maps performance when entities with physics aren't visible. When I think of a game like Vectronic that has clear different areas in 1 map those areas are effecting the performance of the entire map when they don't need to be if he did triggers to show/hide entities and manage the physics in those areas. Hayden's game can be more involved and populated if he's able to control this stuff. We can have highly dense forests where we still have control over each entity to do things with it this way. This just allows for finer control and much better performance from the LE engine. Once of the common knocks I hear of LE is that it's just not very optimized and this can help fight that negative.

 

These are limitations to the number of entities we can have in a map not because of the polycount but because of physics that isn't even being used anyway. This is a limitation that doesn't need to exist. Larger LE maps have been shown to be poor performance by various people over the years and this must be the reason. When these limitations get hit there are no options for people and they get discouraged to continue. If we have this control we now have options.

[Josh]

Yeah but most maps won't contain more than 1000 entities, and you are loading the map up with 100 times that. Everything you add causes some slowdown, but it's not worth trying to optimize things that don't have a big impact.

 

The vegetation system was specifically created because for massive amounts of objects the overhead and memory usage of the entity system eventually becomes inefficient.

[Rick]

The vegetation system was specifically created because for massive amounts of objects the overhead and memory usage of the entity system becomes inefficient.

 

 

Survival games like Hayden's generally give functionality to vegetation (cutting trees, picking berries, etc). With my test I found that I was able to populate a 256x256 meter area with 90% grass and 60% trees and still get 60fps! This was like 8000 entities and it handled it just fine. The problem is when we went beyond that there was a steep drop off to areas of the map I was testing that were hidden because they were far away and shouldn't have affected the area the player was in in any way.

 

I'd say the real culprit to these issues is the physics system. If I recall correctly you seemed to be able to limit physics in your vegetation system in an area around moving entities or something like that? Well, combine with billboard and that type of physics logic the entity system should be able to perform similarly and allow functionality. The veg system as it is still has a place for various grass and bushes, and even the trees if the game in question doesn't allow interaction with the tree, but games these days are just different than they were 5 years ago.

 

When I think back on it this was most likely the issue with Dead Anyway when we were looking a a very modular approach. It most likely didn't have anything to do with the model entity count or polys but the physics entity count that existed on the map when they were hidden (because we managed hiding/showing things in code).

 

The code you have for the veg system for dynamic physics around moving objects would be brilliant to have in the normal entity system and should increase performance even more.

[Haydenmango]

I agree with Rick here that turning off physics on hidden entities or entities that are outside of view range would be extremely helpful.

 

The vegetation system is amazing for non-interactive objects that don't need to be in a specific position like grass, bushes, rocks, and the like but what about buildings or other objects you would see placed in a city? Cities would need specifically placed objects and there could be things like cars that you interact with. In that case having entities turn off their physics when out of view range would be perfect since the vegetation system isn't an option for those objects.

 

Turning off physics on hidden entities or entities outside of view range could be a major benefit to anything that isn't using the vegetation system.

[xtom]

Anything that can boost performance sounds great but I guess a problem with disabling physics on an entity based on view range and hiding it might be for example if you had a physics entity sitting on another entity (like a box on a platform or gun on a desk etc.) and if they both have the same view range and the entity on the bottom hides first it would mean the top entity still shown falls out of position and when both are shown again they would not be positioned as they should be. Maybe it could disable the physics on the entity only if it isn't moving or colliding with a dynamic physics entity but I think most entities will be colliding with something such as floor/walls/furniture except outdoor things on just terrain like veg/rocks so maybe all the extra checks for those kind of situations wouldn't be worth the performance gains.

[Haydenmango]

@xtom That is a very valid concern.

Making it an option on each entity (like the hidden option) to disable physics when hidden/out of view range may be the best idea then to avoid all the checks. Then if you have an object that relies on physics always being active you can keep it that way.

[Rick]

I'm not inclined to do this based on view range but instead on hidden (from us calling Hide() on it). If we explicitly hide an entity it shouldn't be considered in physics. This gives us fine control over things. We can make our own systems based on grid's or triggers to show hide things. However, the idea of only doing physics in an area around a moving physics thing would be great. I think Josh does this for the veg system right now.