In this case my approach could be the following, The enemy character controller should be activated exclusively when in a range very close to the player. In my case I have removed two rows of meshes one on each side of the scene. And only the closest ones have the character controller activated.
This is bringing me to a point of restarting my project based on these aspects of checking performance. And trying to have the best approach so that it doesn't hurt me in terms of FPS.